Identifying the Problem of Gun Control Research

In 2017, anti-gun researcher John Donohue and his team released a study that purported to find that Right to Carry laws increase overall violent crime rates. The paper was updated six months later seemingly to address criticism of its methodology and produced the same conclusion. Readers may be familiar with the study as an anti-gun press fawned over the results without questioning the mechanism of the effect or the methodology itself. 

Economist Carlise E. Moody and lawyer-sociologist Thomas B. Marvell released a critique of the Donohue Right to Carry study recently and concluded that “there is no significant relationship between RTC laws and either murder or violent crime.” 

Moody and Marvell determined that the panel data analysis in the Donohue study is dependent on the use of population weights, which Donohue argues in a response are necessary to avoid implying that the impact of RTC laws on small population states is “equally important” as the impact of the population of the largest states in the Union. 

Despite a spirited argument against Moody and Marvell, Donohue and his co-authors recognize “the difference in results between the population-weighted and unweighted regressions is likely signaling a specification issue.” In other words, the findings depend on the metrics incorporated into the model and the approach used. 

Donohue and his team weighted their data by state population and found RTC laws increase violent crime. Moody and Marvell did not weight by population and found no significant effect of RTC laws on violent crime. The effect is dependent on the weighting. Misspecified models and the misuse of data are two of the most common issues we see in anti-gun research. 

The aspect of the original Donohue study that received the press coverage used a synthetic controls methodology to derive an estimated impact of RTC laws on violent crime. The specific finding is that violent crime is 13%-15% higher after the adoption of RTC laws than it could have been if the state had not adopted RTC. The first question readers may ponder is: how and/or why would RTC laws lead to an increase in violent crime? Law-abiding citizens filling out paperwork, meeting established requirements, and abiding by federal, state, and local laws somehow inspire criminals to commit more crimes? 

Let’s review the reasons according to Donohue, et al.: permit holders may commit crimes they would not have committed without the permit, the culture of gun carrying can promote confrontations, guns may be stolen from people carrying legally, and criminals will find it “increasingly beneficial” to use violence to complete their criminal agenda. Donohue and his team present no more than anecdotal evidence to support these claims. 

Now, consider the synthetic controls methodology. (Where a state is compared to a “synthetic” version of itself created by combining portions of other states). In their first draft, Donohue and his team utilized Hawaii as a contributor to 18 Right to Carry states including Colorado, Kansas, Minnesota, and Nebraska. Donohue revised his approach and included robustness checks in the revised paper, but Moody and Marvell present an additional critique. The synthetic controls methodology could be biased by excluded control variables in both the treatment and control states. 

Much like the specification issue presented in the panel data analyses, the synthetic control methodology is limited by what is missing. That Hawaii was used as a significant component of synthetic Minnesota is amusing, but it really simplifies the missing differences between states. As Moody and Marvell state, “states have certain permanent, or at least long-lived, attributes such as history, tradition, culture, climate, attitudes toward firearms, attitudes toward self-defense, etc., that could affect both crime and the policies adopted to combat crime.” The differences between states should not be overlooked, and restricting the donor pool in any number of ways as a robustness check does not necessarily alleviate this issue – especially when the donor pool of non-Right to Carry states shrinks over the years and more states adopted Right to Carry. 

Moody and Marvell point out that the synthetic controls model cannot control for all relevant factors, pre- and post-treatment, including major factors that cause crime rates to vary. They conclude “the claim that RTC laws increase either murder or violent crime is not supported.” Donohue, et al., present a rebuttal but their original paper helpfully includes charts for the violent crime rate in each RTC state in the appendix. The solid lines are the actual violent crime rate in each state and the dashed lines are the synthetic control version of that state. The vertical line indicates the year RTC was adopted. Go on, take a look. We see more states in which the violent crime rate started to decrease or continued to decrease after RTC than states in which the rate increased. 

But who are you going to believe: the Donohue analysis that overlooks fundamental differences between states or your lying eyes?