A Cautionary Crime Data Tale
A Cautionary Crime Data Tale
Crime data can be poorly reported and inexactly defined so be careful out there!
Crime data can be difficult to analyze in even the best of situations, so it is important to be cautious when using crime data to ensure that what your analysis says matches the data. It's best to ensure that the incredible surge in crime you’re describing is not being driven by a change in reporting practices and sometimes crime is falling precipitously simply because an agency did not report full data that year.
One recent story that probably needed a bit more data caution carries the headline “NRA-Style Politics Transformed Canada’s Gun Culture — and Shootings Rose 869%”. Before I start on my complaints with the data, I do want to note that the story is a good piece with rich details about an important issue. The piece is deeply reported and produces compelling evidence that an influx of US guns is contributing to increased gun violence in Canada, but I found it wanting from a crime data perspective.
The headline of an 869% rise in shootings, however, is much more suspect and provides a cautionary tale for being skeptical about reported crime data. One issue I have is that “shootings” in the piece are defined by the authors as being synonymous with the Canadian crime of discharging firearm with intent. One should always be cautious when ascribing one's own definition to a piece of crime data that might not match the original offense.
The main problem is that shootings as most people think of them — an incident where somebody is hit by a bullet — are not inherently synonymous with firearm discharges as is implied in the piece. I would certainly advise against using the terms interchangeably when analyzing crime trends.
The Toronto Police Department has a dashboard that more or less nicely sums up the definitional differences:
Shooting: Any incident in which a projectile is discharged from a firearm (as defined under the Criminal Code of Canada) and injures a person. This excludes events such as suicide and police involved firearm discharges.
Firearm Discharge: Any incident where evidence exists that a projectile was discharged from a firearm (as defined under the Criminal Code of Canada) including accidental discharge (non-police), celebratory fire, drive-by etc.
Now discharging firearm with intent as used in the piece is a more specific subset of firearm discharges, but the Toronto police definition drives home why they shouldn't be used interchangeably. Shootings where a person is hit are reported to the police with nearly complete accuracy while firearm discharges are massively underreported to the police. Using only instances of firearm discharge that were successfully reported to the police would be highly inaccurate stand in for shootings, and this methodology prone to misinterpreting changes in reporting practices with changes in gun violence.
Jennifer Doleac and Jillian Carr have done terrific research which highlights the underreporting of firearm discharges. They used 911 data and ShotSpotter data to estimate how often actual gun violence events were successfully reported in Oakland and Washington, DC. They found that “only 12% of gunfire incidents result in a 911 call to report gunshots, and only 2-7% of incidents result in a reported assault with a dangerous weapon.” In other words, only about 1 in 10 firearm discharges were reported to the police and about 1 in 20 had an actual report written by the police acknowledging the crime. This research matches Calls for Service in New Orleans where only 7 percent of shots fired incidents this year have a Report to Follow disposition.
The meteoric rise in discharging firearms with intent in Canada fails the logic test as an actual enormous rise in gun violence. Consider Saskatchewan, called the “home to Canada’s highest rate of gun crimes” where discharging firearm with intent offenses rose over 1,100 percent between 2004 and 2017 while homicide fell by 3 percent. Discharging firearm with intent offenses rose by 326 percent in all of Canada between 2005 and 2016 while the number of firearm homicides fell by one (sadly firearm homicides can't be broken down at the geographic level). It makes no sense that the number of times people were shot at would more than triple but firearm deaths would fall.
The thing is that there is a story of rising gun crime in Canada in the data, especially over the last few years, but focusing on an 8-fold increase in discharging firearm with intent offenses masks the scope of the real story. Other gun crimes — such as firearm homicides, use of a firearm in commission of a crime, and pointing a firearm — have uniformly risen since 2019 though the rise has been nowhere near as catastrophic as what is suggested by the discharging firearm with intent data.
The Toronto Police Department provides data on shooting victims with an injury (or death) and firearm discharges which highlights how unreliable the latter is. While shootings with an injury or death bounced around in Toronto between 2004 and 2022 the number of firearm discharges (firearm incidents without an injury or death) has surged. There were fewer shooting victims in 2022 than there were in 2005 in Toronto, but there were nearly 230 percent more firearm discharges reported in 2022 than in 2005.
There were 4 times more shootings with an injury or death in 2006 than firearm discharges in Toronto suggesting the firearm discharge numbers for the early 2000s are fairly worthless. Toronto’s homicide rate has largely kept pace with the city’s population growth — the city's homicide was rate lower in 2020 than it was in 2004 — while the rate of discharging firearm with intent offenses was over 9 times higher in 2020 than 2004. That only makes sense as a change in reporting practices.
A final reason to be cautious about the firearm discharge data is that the data collection early in the 2000s may be more incomplete than more current data. I am not an expert in Canadian crime data - gasp, I know - but the Canadian Centre for Justice Statistics provides a handy methodology of data collection each year. In 2003, the survey’s methodology section notes:
In 2003, 122 police services in 9 provinces supplied data for the complete year to the UCR2 survey. These data represent 61% of the national volume of reported actual (substantiated) Criminal Code crimes.
In 2022, by contrast:
In 2022, about 150 police services in all 10 provinces and 3 territories supplied data for the complete year to the UCR2 survey and represented approximately 100% of the population of Canada.
I do not know whether this methodological issue makes a huge difference in reporting figures in those years. But the methodological change, the squishy definition of shootings, and the data not passing the logic test are all huge red flags in how the trend could and should be reported. A less breathless statistical claim thanks to data uncertainty would have improved, rather than taken away, from what was otherwise a solid and important piece.