Does Falling Murder Simply Reflect a Methodological Change?

Short answer: No. Longer answer below.

One of the most common responses I hear from people pushing back on the belief that murder fell in 2023 was summed up in an email I recently got. The sender said this:

I have been referring to your analysis of the possible decline in murder rates in 2023.

The retort I receive is based on the Marshall Project's position that a 2021 change of system and methodology led to incomplete data reporting and make the result inconclusive, if not in fact very flawed.

Can you help sort this out for me, or refer me to something that might?

I see this a lot, and I don’t believe it is an argument that is made in bad faith — unlike many other arguments I do hear on this subject. The Marshall Project has indeed written extensively on the FBI’s transition from the Summary Reporting System (SRS) to the National Incident-Based Reporting System (NIBRS) that took place in 2021 (as have I).

The gist of the change was this: agencies have reported offense data covering seven major categories of crime since the 1930s. The FBI tallies these offenses and puts out an annual report that gives national estimates. These estimates are formed from roughly 95 to 97 percent of around 18,000 agencies reporting data to the FBI each year.

In 2015 the FBI announced that it would no longer accept data through SRS and would only allow agencies that were NIBRS-compliant to report data in 2021. The advantage being SRS has just seven offense categories whereas NIBRS has dozens.

But only around 65 percent of the US population was covered by an agency that was NIBRS compliant in 2021, so the 2021 estimates were a nightmare. The argument that methodological changes impacted our ability to understand crime trends certainly holds merit with respect to the 2021 estimates. Those estimates were less reliable and came with quite large error bars — an discussion that people like me were making at the time.

There are two main reasons for why this argument is incorrect when applied to evaluating the current trend though.

First, and this may come as a shock to some people, it is no longer 2021.

The national estimates for 2022 were much better because the FBI allowed non-NIBRS agencies to still submit data under SRS. Crime data was available for well over 90 percent of the US population in 2022 compared to around 65 percent in 2021 thanks to this reporting change.

So while it would be appropriate to put an asterisk around some of the national crime figures for 2021 (I think the murder estimate is pretty reliable given how closely it matches what we would expect from a large city sample), no such warning is necessary for the most recent data from the FBI.

If someone wants to argue that 2021's national estimates for all crimes should be treated as suspect and we should only compare 2023 to 2022 or to 2020 (or earlier) then I won't disagree. But it is incorrect to suggest that the NIBRS switch is still hampering comparisons for 2022 and 2023.

I've also heard the argument that the quarterly estimates from the FBI should be thrown out because New York City, Chicago, and Los Angeles aren't included. I find that rationale unconvincing because a) the quarterly estimates were pretty good at establishing the national trends in 2020 without those three cities, b) around 80 percent of the nation’s population is represented which is a good enough sample to make an accurate albeit imprecise estimate of the trend (which is all we want), c) the quarterly estimate mostly matches the YTD murder sample that we collect which does include those cities, and d) inclusion of those cities shouldn't be expected to change the overall trend that much given that murder fell quite a bit in all three while overall crime was about even in NYC and LA while it rose in Chicago largely on the back of a surge in auto thefts.

The second reason that the NIBRS transition isn’t responsible for the crime trends we are seeing is because our YTD murder dashboard relies on a completely different methodology than the one employed by the FBI to calculate national estimates.

Ours is a sample of data from cities with publicly available data, so we get the data straight from the city source rather than waiting for FBI's data release. Obviously 18,000 agencies of data is much, much more robust than 200 or so, but the sample size is sufficient for its purpose: describing the trend we would expect to see from the larger set of agencies as best as possible in as close to real-time as possible.

If one wanted to argue about the weaknesses of the sampling methodology then one could point out that it is a preliminary count and a considerably larger sample size would be helpful for getting the margin of error down to one percent or so. As it stands, this sample *should* be within two or three percent of the FBI estimate based on historical patterns. But neither of those quibbles are impacted by the NIBRS switch, and neither changes the bottom line of what the methodology finds: a large decline in murder happened across much of the nation in 2023.

I get that crime data can be confusing to navigate, and it is unfair to expect people to understand the nuances between SRS and NIBRS data collection and reporting. That said, the methods used to evaluate the crime trends in 2023 are not reliant on NIBRS-specific data and are not comparing a year of solid reporting to a year of sketchy reporting.

These methods are used not because they are perfect but because crime data is slow and imperfect data can create a reasonably respectable understanding of trends. If we appropriately caveat our uncertainty derived from imperfect sources then the data is incredibly useful for helping to evaluate where we are in the context of where we've been. That's how good analysis is done.