Attempting To Audit The Quarterly Data

Taking a look under the FBI's Q1 data hood.

Violent and property crime were not down 15 percent nationally in the first quarter of 2024 relative to the first quarter of 2023. I don’t assert many things in this newsletter but I’m going to assert that, and I wrote a fair amount two weeks ago about why. Violent and property crime were likely down a healthy amount in the first quarter, but the available evidence suggests it was a considerably smaller decline.

The first thing to remember is that Q1 2024 was the first time the FBI has ever produced quarterly data for the first 3 months of a year. It's a new product and learning about its reliability will take some time.

The first quarter data is more prone to data errors than later quarters because agencies have more time identify and correct data errors as the year goes on. Sometimes agencies fail to report complete data for a month directly to their state UCR program, and so the data dutifully submitted to the FBI is wrong. There’s time built in to identify there reporting errors and fix them before the final data submission deadline (though it isn’t always done).

Consider the case of Dallas — a small town in Texas outside Fort Worth. The Dallas Police Department reports data to the Texas Department of Public Safety on a monthly basis usually a week or two after the most recent month has ended. The data reported in April for March 2024, however, was a pretty severe undercount punctuated by just two murders being reported in Dallas that month (there were 16 per the police department). I took the below screenshot from the Texas DPS’s excellent Offense Trends Report sometime in May to highlight an obvious data submission error in the wild for a presentation.

But when the agency submitted its data for April or May it added the March missing data. If you run the same report now here is what you get:

The system works!

But if one was putting together quarterly data and using data submitted prior to the correction then one might have a pretty substantial undercount of that agency’s crime figures. Which is what happened with the FBI reporting 1,300 violent crimes in Dallas in the first quarter of 2024 compared to the 1,873 reported now by the Texas DPS.

I tried to repeat this exercise with as many cities with public data to better understand the national trend through Q1. I found publicly available data on violent and property crime through the first quarter of 2024 compared to the first quarter of 2023 for 112 of the 231 cities above 100,000 that reported data to the FBI in the first quarter. I did not include New York City or Washington DC in this sample because neither city reports public data to UCR Part I standards (NYPD reports felonies and DC reports to their criminal code which means their public data undercounts aggravated assault).

The below graph compares the percent change in quarterly violent crime as reported by the FBI for each agency (horizontal axis) versus the percent change reported publicly by the agency (or state UCR program) (vertical axis). Cities above the gray line in are places where the FBI’s reported percent change is larger than the change reported by the agency (and vice versa for below the line).

There are quite a few more cities above the line (crime was underreported by the FBI) than below the line (crime was overreported) highlighting the strong likelihood that crime overall was underreported in the first quarter. North Las Vegas stands out as reporting 334 violent crimes in Q1 2023 compared to just 36 in Q1 2024 while the Nevada state UCR program shows a considerably smaller (but still quite large) decline through March.

Still, violent crime was down in 67 percent of the cities that reported both to the FBI and publicly through the first quarter. Violent crime was down 7 percent in the 112 city sample that I was able to check and property crime was down 8 percent. Those are large declines which, on top of likely declines in 2023, suggest that crime continued to fall at a strong pace through the first part of 2024.

But the FBI had violent crime and property crime down 11.6 percent and 11.4 percent respectively in those same cities and a 15 percent decline in both categories nationwide. So while it’s not true to say that violent and property crime are each down 7 to 8 percent nationally, this exercise adds credence to the idea that the FBI’s estimates are off by a decent amount.

The quarterly data is not perfect, but having it is much better than not having it. Most agencies will provide reasonably close crime counts compared to their actual total each month that they submit, but a handful will occasionally be off over the course of a year. This is less of a big deal in the third and fourth quarters when a single month of underreporting is more diluted than it is in the first quarter of the year.

This effect is apparent when comparing the change in murder over time in the third and fourth quarter last year. Murder was down 12 percent in our city sample through September 2023 and it was down almost 12 percent through the end of the year, but the change in murder nationally per the FBI went from down 16 percent through the third quarter to down 13 percent through the fourth quarter. My guess is that the movement from -16 percent to -13 percent largely reflects more agencies finalizing their data.

Critically, the overall change in the first quarter data is large enough that it doesn’t impact the assessment of the big picture trend that murder is down a lot and violent/property crime is down a fair amount. It does, however, add uncertainty into exactly what the final year tally might be given that the data is flawed and there are still a lot of time left in the year. It also serves as a reminder that this information is preliminary and the exact reported figures should always be taken with a grain of salt.