Another Cautionary Crime Data Tale
The fourth entry in the popular (?) series.
A somewhat recent piece in The Dispatch titled “Why the Perception That Crime Is Rising Persists” discusses national crime data though there are numerous inaccuracies. My newsletter is not to here to shame folks who misuse crime data nor is it to engage in partisan persuasions. So I’m going to highlight the ways in which the author incorrectly uses crime data in the piece in a way which hopefully makes others aware about the potential pitfalls of working with this information.
Let’s dig in! From the piece:
“NIBRS and Uniform Crime Reporting (UCR) data are the primary sources of crime data in the U.S. Compiled by the FBI’s Criminal Justice Information Services (CJIS), the 18,000 or so law enforcement agencies from around the country send their crime data to CJIS annually. This data is curated by the sending agency and is admittedly incomplete. Crime data is measured both in raw numbers and per 100,000 persons. Regardless of the inconsistencies, UCR and NIBRS remain the gold standard of crime data in the U.S. But some analysis to Biden’s claims is warranted, especially given the persistent perception that crime is rising.”
This paragraph is mostly correct, but there is a subtle problem with how NIBRS and UCR are referred to which I think forms the root of the issue. NIBRS and UCR are not separate entities, NIBRS is a subset of UCR.
Traditionally, agencies have submitted data via the Summary Reporting System (SRS) which goes back to the 1930s. The FBI publishes an annual compilation of statistics from SRS called the Crime in the US report. NIBRS — which is more expansive — was created in the 1980s to collect more robust data. Agencies were expected to transition to NIBRS by the end of 2020. Many didn’t, it was a big problem, I’ve written about it a bunch.
The problem mostly got solved in 2022 and NIBRS participation was even better in 2023. But NIBRS and SRS are two peas of the same UCR pod just as the American League and National League are both parts of Major League Baseball.
UCR is the gold standard of crime data — manyfold warts and all — not NIBRS. And when you think about violent crime you’re probably thinking about murder, rape, robbery and assault which are SRS — rather than NIBRS — violent crimes.
“To start, NIBRS and UCR data does not go back 50 years, so scoring the claim on that basis would be nearly impossible. Moreover, the 2023 quarterly data is not the full complement of data compiled in the annual data sets, which is only as current as 2022. Arbitrarily examining the last 25 years will suffice. Also, in fairness to the president, keep in mind that he cited the “crime rate” and not the raw number of violent criminal offenses. It’s a distinction with a huge difference and possibly the foundation of the persistent perception that crime is rising.”
Oh no. I think I see what’s happening here. NIBRS does not go back 50 years. UCR goes back more than 90 years. Here it is in 1930. And 1953. And 1970. There are certainly challenges to analyzing the specifics of older crime data, but to argue that it does not exist going back 50 years is wholly incorrect.
“In 1998 (25 years’ worth of data ago), the violent crime rate was 567.6 per 100,000 persons, the highest rate in the last quarter century. By comparison, the same FBI data source indicates that the violent crime rate in 2022 was only 380.7 per 100,000 persons, or about a one-third reduction in the violent crime rate in that timespan. Any visual evaluation of the FBI graph below might indicate that we are safer now since the crime rates in America have steadily declined over the past 25 years.”
No issues with this paragraph, but it's worth noting that the underlying graph and analysis is coming from SRS data.
“This is where looking under the hood of the data becomes useful for real crime analysis. In fact, the raw number of crimes is much higher.
In 1998 there were 119,724 violent criminal offenses reported. Compare that to 2022, when there were 951,270 such offenses reported. That is a staggering 695 percent increase in violent criminal offenses over that 25-year period. There were approximately 894,000 violent criminal offenses reported in 2023.”
Now we’ve hit the crux of the problem and why this analysis of the data is so deeply, deeply flawed. The author is now using NIBRS offenses for 1998 and then using the quarterly estimated percent change for SRS violent crime in 2023 and applying it to the number of NIBRS offenses reported in 2022.
This cannot be done.
To begin with, only 2,361 agencies were using NIBRS in 1998 covering 10 percent of the US population. The CDE isn’t great, but if you pull up the NIBRS data it immediately tells you this fact.
In 2022 that figure had jumped to over 13,000 agencies covering 75 percent of the US population. One cannot use 1998 NIBRS totals to compare to 2022 NIBRS totals. One cannot use 2020 NIBRS totals to compare to 2022 NIBRS totals. One cannot use 2022 NIBRS totals to compare to 2023 NIBRS totals. The population of reporting agencies is changing so rapidly as to make any comparisons of NIBRS offenses from one year to the next completely invalid. If you want to compare NIBRS offenses over time in an agency or group of agencies that have been NIBRS-compliant for the whole timeframe then go for it.
One can use SRS which has been largely unchanged over the last 60+ years — other than 2021 — to compare violent crime1 totals and rates over that span.
Rates are very useful because the US population has grown a lot over the last 60 years.
In addition, you can’t just use a 6 percent estimated decline in the quarterly estimates and apply that to NIBRS totals for 2023. The number of NIBRS agencies will be higher in 2023 than in 2022 which will inevitably mean more reported crime. Also, SRS violent crimes are not synonymous with NIBRS violent crimes, there are differences in each category. It's all just a comparison of apples to footballs and it makes no sense.
“Moreover, if 2020 is stripped from the data set, Joe Biden has presided over the three deadliest years in American history … and it’s not even close. The celebrated 2023 was one of the most violent years in American history. Violent crime offenses are generally higher under Biden as well. For instance, in 2020 (the most violent year during Trump’s administration) there were 673,812 violent crime offenses reported. By comparison, Biden’s storied 2023 saw nearly a quarter million more violent criminal offenses.”
This paragraph nearly broke my brain. The vast majority of violent crimes are not deadly, so even if the stretch of 2021 to 2023 featured the most violent crimes in American history (which it did not) that would not inherently equate to the most deadly years (also we’re forgetting about the Civil War and whatnot here).
Just using murder stats shows 2021 to 2023 had about 10,000 fewer murders than were recorded between 1991 and 1993 (about 73,000 vs 63,000) in a country with 70 million or so more people. Murder totals and rates being below where they were in the 1990s certainly does not excuse the surge we saw in 2020 and 2021, but neither does it allow for hyperbole about the deadliest years in US history (also, 23,000 people were killed in one day at Antietam!).
“In 2022 there were 2,076,644 arrests in the U.S. In 2021, there were 1,762,945. By comparison, there were more arrests made during Trump’s administration with the exception of 2020, when there were 2,051,465. In fact, law enforcement made more arrests in 2020—when the nation was literally locked inside—than in all of 2021.”
The final paragraph with glaring errors in using crime data is the one above which gives us the chance to talk about arrest data. There were not around 2 million arrests in 2022, there were around 6.2 million. There were 2,076,644 arrests categorized by the FBI as “All Other Offenses (Except Traffic)”. The same is true of the 2021 and 2020 figures mentioned in the data above.
In addition, the 2021 figure is substantially effected by the NIBRS switch which led from 15,600 agencies reporting arrest data in 2020 to 13,100 in 2021 (and back up above 15,600 in 2022).
The thing is that arrests did fall between 2019 and 2022 but this is likely largely due to the effects of the pandemic accelerating a trend that began more than 15 years ago.
Arrests fell every year from 2007 to 2020 and there were 37 percent fewer arrests nationally in 2019 than in 2009. Yet the nation’s violent crime rate was lower in 2019 than in 2009.
Crime data is challenging to use even in the best of contexts. It often caries caveats and reporting issues that makes precision impossible. Acknowledging and leaning into these faults is the only way to analyze the data with accuracy. Hopefully this newsletter shows the pitfalls that come with charging into crime data without carefully accounting for where the information is coming from and how comparable it may be to previous years.
I’m using the legacy definition of rape to make violent crime rates from 2013 to 2022 comparable to previous years.
Conservatives have it in their brain that Crime is out of control !!! they run on a pure reactionary drive that must manufacture the crime they imagine. When compared to other countries the death toll in America is wildly high. Americans should be more shocked at how bad our health outcomes are and the expensive they are compared to the world. highest child birth mortality rate, 100K dead every year from Drug OD's, 50K a year die from lack of healthcare, 40K are dead from guns and 1/2 of those are suicides... for fun I post clips of your articles on Next-door, and you would not belive the amount of people who respond! Crime is not lower a guy I know just got car jacked and the police man was shot 2 weeks ago in a different city last week....
I think your analysis here is persuasive.
The one tangential concern I have (which does not undermine your points here) is that I still haven't seen any good reason to trust the stats on most crimes besides homicide and auto theft, crimes which are subject to substantial undercounting or subjective determination of degree of offense or even whether any offense occurred at all. The incident where San Francisco shoplifting incidents doubled from one month to the next simply because a single Target started calling in all of their incidents is illustrative. It seems obvious to me that citizens' interest in reporting crimes would vary depending on both how aggressively law enforcement pursues such reports and how inconvenient law enforcement makes it to take a report to start with, and it would vary in a noisy but continuous way, so I would expect every little change to have some little effect on underreporting rates once the public gets the message.
(The NCVS victimization survey, which could serve as a check on police reports, is filled with questions that start out fairly well defined before lapsing into vagueness -- by asking about "attempts," for example, or specifying that incidents of unauthorized borrowing by household members are to be counted as theft -- which may or may not be criminal and in any case makes it very hard to objectively count discrete incidents. "Sis borrowed the car without permission again" does not deserve to be tallied in the same slot as "My car disappeared off the street and its gutted burnt remains were found two months later." I'm not saying such situations should be ignored, but including marginal and uncompleted crimes is detrimental to trying to get a reasonably objective and comparable crime count.)
I wonder whether the divergence between crime rates and homicide rates from the mid seventies to the late nineties was real (homicide rates plateaued, crime rates kept rising). I don't know, but the well-known more aggressive enforcement of the nineties could have just led to fewer undercounts and fewer offenses dropped into lower categories.