US Crime Data Reporting Remains a Mess

More agencies publishing more open data could help alleviate the problem.

The public lacks consistent, accurate, and actionable crime data that is critical for our ability to identify new trends and evaluate solutions, and it is unclear how long it will take for this problem to get fixed. The share of agencies reporting data to the FBI’s Uniform Crime Report (UCR) program has plunged producing real costs for our ability to effectively use crime data. Agencies should take advantage of this moment to implement 21st century data practices to improve both internal analytics and external transparency.  

The first UCR bulletin was published in January 1930 covering data from 400 cities and 43 states. By 2020 there were nearly 19,000 law enforcement agencies covering roughly 97 percent of the US population reporting annual data to the FBI under the UCR Summary Reporting System. Detroit police chief William Rutledge cogently summarized the impetus for UCR in 1927, saying “In the past few years the control of crime has been an absorbing topic before the American public. It has been taken for granted by many that a crime wave exists…[N]either the American public nor its regularly constituted authorities responsible for police, court and correctional activities have any reasonably complete and accurate knowledge of the amount and cost of crime in this country, or whether it is increasing or decreasing…All of which means that we are perforce, in the absurd position of endeavoring to diagnose and cure a social disease with little knowledge of its causes, its nature and its prevalence.”

Rutledge’s words are nearly a century old, but they are nearly as true today as when they were spoken.

Americans perceive that crime is increasing today, just as they did a century ago, even as major crime as reported under UCR has fallen for most of the last two decades. Gallup recently found that 78 percent of Americans believed crime has increased nationally over the past year and 56 percent said crime had increased in their area. Crime became a major talking point during the recently concluded midterm elections and was discussed everywhere from stump speeches to debate stages.

Murder – unlike property and violent crime as a whole – has risen dramatically over the last three years almost certainly fueling some of the public’s misperceptions about crime. Murder rose 29.4 percent from 2019 to 2020 according to the FBI, the largest one-year increase both in terms of percent and number of additional murders. Murder was up in big cities with 1 million or more people, and it was up in small towns with less than 10,000 people. It was up in the suburbs, in rural counties, in places that voted for Biden and in places that voted for Trump.

The rise in murder has also been accompanied by a precipitous decline in agencies reporting crime data to the FBI resulting in a situation not unlike what Rutledge described in 1927 where our ability to describe, understand, and counteract this new trend is severely diminished. In 2015, the FBI announced that all agencies would be required to report via the more expansive National Incident-Based Reporting System (NIBRS) starting in 2021 or they would not be able to report anything at all to the FBI.

Only 31 percent of the US population was covered by a NIBRS-reporting agency when the decision to switch to mandatory NIBRS reporting was announced in 2015. The good news is that coverage more than doubled between 2015 and 2021, but the bad news is that far fewer agencies are reporting data to the FBI today than a few short years ago. Roughly 64 percent of agencies reported data via NIBRS in 2021, far below the 97 percent that typically reported data each year under the old SRS system. What’s more, only 52 percent of agencies submitted 12 months of data covering the full year in 2021. Coverage has improved in 2022 though it remains far short of where coverage under SRS a few years ago.

The result is substantially more uncertainty with regards to crime trends. The FBI produced a trend analysis in October estimating UCR Part I crime (murder, rape, robbery, aggravated assault, theft, auto theft, and burglary) in 2020 and 2021. Underreporting, however, required reporting confidence intervals for both 2020 and 2021 estimates. The FBI wrote that “In most instances, differences in the comparison of 2021 data to 2020 do not meet the criteria for statistical significance.” In other words, most estimated increases or decreases in crime from 2020 to 2021 fit within the confidence interval where we cannot be certain whether they indicate a real increase or decrease.

Murder, for example, was estimated to have increased from roughly 22,000 in 2020 to 22,900 in 2021 (the estimates were rounded to the nearest hundredth). But the confidence interval for 2021 was somewhere between 21,300 and 24,600 murders while the confidence interval for 2020 was between 21,000 and 23,000 murders. So, while the FBI estimated a 4 percent in murder, murder could have risen 7 percent, or it could have fallen 17 percent and either would fit within the FBI’s estimated confidence interval and there will never be a way to say with certainty what happened.

Estimates of property and violent crime produced even larger confidence intervals. The confidence interval for violent crime was between a 12 percent decline and an 11 percent increase while the confidence interval for property crime was between a 38 percent decrease and a 50 percent increase!

There are also clear problems with estimations provided in the trends analysis. The state of New York, for example, is estimated to have had between 8,900 and 31,500 violent crimes in 2021 which is well below the 76,229 violent crimes the New York Division of Criminal Justice Services reports actually were recorded that year.

Estimates are also rounded to the nearest 100 which makes some trends difficult to decipher. For the crime category Sexual Assault with an Object with Firearm Used the FBI estimates there were 100 offenses in 2021 with a confidence interval stretching between 100 on the low end and 100 on the high end. In 2020, by contrast, the FBI estimates there were 100 offenses with a confidence interval between 100 and 100 offenses all of which adds up to an estimated 28.8 percent increase in these crimes.

NIBRS presents an excellent opportunity for academics and researchers to dive deeply into crime and policing topics like never before. Additionally, there are plans to have non-fatal gunshot victimization be reportable by NIBRS starting in 2023 which could create the first national record of non-fatal shootings. But NIBRS in its present form is virtually useless for evaluating and communicating national crime trends to a concerned public.

So, what can agencies do about it?

The most important step agencies could take if they are not NIBRS compliant is to become NIBRS compliant as quickly as possible. The 2021 NIBRS report was produced without any data from New York and Los Angeles police departments. Baltimore (MD), New Orleans (LA), and Jackson (MS), cities which likely led the nation in murders in 2022, also are not yet participating in NIBRS. Chicago and St Louis submitted a few months of data in 2021 – though St Louis was mistakenly credited with a full year.

But the problem goes far beyond just big cities. While 7 of the 19 agencies covering over 1 million people did not report, there were also over 3,500 small agencies representing populations under 10,000 people that reported no data accounting for nearly 40 percent of all such agencies. Only 2 agencies in Florida, 15 agencies in California, and 31 agencies in Pennsylvania submitted any data to the FBI covering crime in 2021.

Reaching national reporting thresholds analogous to what was achieved under SRS is essential for being able to honestly evaluate crime trends without massive confidence intervals.

Another step agencies can do – regardless of whether they are NIBRS compliant – is to embrace open data as a novel method for informing the public about crime trends. Open data need not be complex. Many agencies post crime statistics in easy to digest formats on their websites and update them daily, weekly, monthly, or quarterly.

This type of information can be compiled to form a baseline estimate of crime trends in the United States without waiting months for NIBRS data to be assembled and reported. We use regularly reported data from 90 or more agencies to create a real-time estimate of murder trends in the current year relative to the previous year.

This methodology has some drawbacks, especially early in the year when counts are low and percent changes can have large swings. But it also provides a reasonably accurate understanding of what to expect from national crime statistics in a fraction of the time and at virtually no cost. There is also room for growth. As more agencies produce data then the estimated changes can become more accurate.

The value of open data goes far beyond aggregated crime statistics. Most of the posts you’ll see here will rely heavily on crime and policing open data. Many agencies produce daily data on Calls for Service, reported offenses, use of force, misconduct complaints, stop and search, recruitment and retention, and much more. These data sources allow the public to understand how policing is being performed by their home agency, provide insight into important topics such as crime trends and response times, and enable law enforcement agencies to communicate their activities more effectively to the population they serve.

A handful of agencies in cities like Cincinnati, Philadelphia, and Rochester (NY) publish daily data on shooting victimization. This kind of information is critical for understanding an unprecedented rise in murder that has largely been fueled by gun violence, but only a precious few agencies nationwide make this information available to the general public.

It will be years before enough agencies have converted to NIBRS to make large estimates a thing of the past. Completing this process is an absolute must, but in the meantime people who work with NIBRS data should acknowledge and embrace the uncertainty that inherently comes from estimating crime statistics for a large portion of the country. And while we wait for NIBRS to catch up, agencies should publish both anonymized raw data and aggregated crime statistics enabling interested parties to evaluate trends without having to wait nearly a year for flawed estimates to be released.