Suspicions About the SORNA Challenge

Apparently something called the National Institute of Justice is offering prizes for whoever comes up with exciting and innovative ways to study cost and effectiveness of the Sex Offender Registration and Notification Act (SORNA). The appellatesquawk quotes from the breathless email announcement:

NIJ CHALLENGE: COST-BENEFIT OF SEX OFFENDER REGISTRATION LAW

Are you up for the challenge? Enter NIJ’s first-ever SORNA Challenge! NIJ is seeking innovative ways of developing strategies to measure the implementation costs and public safety benefits of the Sex Offender Notification and Registration Act (SORNA) – part of the Adam Walsh Child Protection and Safety Act of 2006 – by improving the effectiveness of sex offender registration and notification programs in the United States.

…

A cash prize of $50,000 is available. Deadline: Oct. 31.

The announcement also makes this important observation:

Empirical research on sex offenders has grown over the past decade, but no study to date has examined the multifaceted effects of SORNA, specifically the wide range of costs that have been or may be incurred in implementing SORNA, or the public safety benefits achieved with SORNA compliance.

The squawk is a bit upset by that revelation:

This is apparently a new streamlined process of how a bill becomes a law: first enact it, then figure out what it costs and what it does.  What next? Cash prizes and free stuff for figuring out the cost/benefits of the Iraq war?

Given the dreadful aspects of sex offender registration laws — the limits on where registered offenders can live and work, requiring prostitutes and horny teenagers to register as if they were child molesters, the near-impossibility of a return to normal life — I understand his outrage. But to be fair to the NIJ, you can try to predict the effects of legislation in advance (with varying degrees of success) but you can’t actually measure those effects until the legislation is in place.

Even then, measuring the effects of legislation such as SORNA is difficult work. The worst approach — all too common in some circles — is to measure something like the number of offenders registered, or the rate of compliance with in-person check-in requirements, or the number of times the public queries the registration database. The are relatively easy numbers to obtain, but they are the wrong numbers: They are inputs, and we want to measure results. Looking at those numbers would be like measuring the effectiveness of an advertising campaign by counting the number of ads placed instead of the amount of product sold.

So what we want to measure is the effect that SORNA has had on the crime rate. The problem is that while measuring the rise and fall of crime rates is relatively easy (although there are complications), it is much harder to determine why the crime rates rise and fall. There are many causes, all operating at the same time, yet we are only interested in one of them. We need to find a way to untangle the effects of SORNA from all the other things that affect crime rates — demographics, the economy, migration, public safety budgets, police tactics and equipment, and so on. This is a very complex problem, filled with opportunities for error, but social scientists have developed methodologies that sometimes give meaningful results.

I’m no expert, but I think I can describe the general idea behind these methods: The SORNA act works by imposing requirements on local law enforcement agencies, and those agencies have been coming into compliance separately at different times since the law was passed. Furthermore, some of these agencies were already in compliance with parts of SORNA before it even passed. This gives us a kind of natural experimental randomization, since SORNA came into effect in different places at different times, it is unlikely to correspond by coincidence with other factors affecting the crime rate. In one city SORNA might have taken effect in combination with an increase in police manpower, while in another city at another time, SORNA might have taken effect while police manpower was decreasing.

Given a large enough amount of data, it might be possible to detect the effect of SORNA. Perhaps we could look at crime rates in every agency’s territory for, say, the three years before and after SORNA requirements were implemented. On average, if SORNA is providing benefits, we should see a greater reduction in crime (or a smaller increase) for those periods than for comparable periods at agencies that did not implement SORNA.

In addition, some of the confounding factors are measurable, such as the local economy and police budgets — and we can statistically estimate the effects of these factors on the crime rate and then subtract them out of the data to compensate for their effects. This should more clearly bring out the effects of the SORNA legislation.

This kind of study is a complicated undertaking, requiring obtaining SORNA compliance and crime rate data from as many states as possible, and ideally obtaining at least some data on a county-by-county or agency-by-agency basis. Then all the data has to be converted to a single coding system that can be fed into statistical analysis software. There could be lag effects (maybe the benefits of SORNA don’t show up until two years after implementation),and the data might not be robust (in some studies of the deterrent effects of the death penalty, the effects go away if you eliminate Texas from the data). And because I don’t do this sort of thing for a living, there’s probably lots of other issues to worry about that I’m not aware of.

However, in general, that’s the methodology for studying this kind of effect. This work is not easy, and the results are often of poor significance, but the process and its limitations are fairly well understood. It’s a standard approach to these kinds of studies, widely used and accepted by sociologists, economists, and public health statisticians.

Which raises the question…what is it about the accepted methodologies that causes the NIJ to seek “innovative” approaches.

A cynic might suggest that the normal methods are insufficiently rigged in SORNA’s favor. Consider this part of the request for proposal:

NIJ’s SORNA Challenge seeks creative and innovative research strategies for future researchers to use when studying (1) the implementation costs associated with complying with SORNA and/or (2) SORNA’s public safety benefits (examples include, but are not limited to, the Act’s general and specific deterrent effects, its effect on law enforcement’s ability to prevent crime, and its effect on the public’s ability to protect itself).

What stands out immediately is that these two challenge components add up to a highly biased benefit-cost calculation: The study proposal uses a broad definition of the the public safety benefits of SORNA, but limits the cost component only to implementation costs. This totally ignores the costs incurred by anyone outside the government, especially the costs incurred by people whom the government wishes to place on such registries, such as the legal costs of fighting registration, the loss of employment, and the general loss of freedom to live where they want and do what they want. Basically, they’re counting only part of the cost, but they’re including all of the benefits.

Allow me to illustrate the problem with an absurdity: Suppose that instead of SORNA, we were to pass RAAMSA, the Random African-American Male Shooting Act. RAAMSA funds local law enforcement programs in which inner-city police officers annually gun down 10,000 young black men at random. Using the methodology implied by the SORNA Challenge, this is a highly cost-effective program: Assuming $2500 to find, shoot, haul off, and cremate each body, the annual budget is only about $25 million dollars. Given that the lifetime incarceration rates for black men is about 30%, we can therefore assume that RAAMSA takes at least 3000 criminals off the street every year at a cost of less than $8500 each — far below than the cost of imprisoning those criminals for even a single year, and RAAMSA is a permanent (or should we say final) solution.

If the SORNA Challenge study components would miss such a monstrous evil as this, I think it’s safe to say that they could create a misleadingly rosy impression of the benefit-cost qualities of SORNA.

Now it’s entirely possible that no one intends to use these components in a misleading way. Perhaps, years from now, everyone involved with this study will freely and honestly admit its limitations. Perhaps no one will look at the results of these two studies and conclude that the benefits of SORNA outweigh the costs just because the benefits of SORNA outweigh the implementation costs of SORNA. Perhaps no one will issue pronouncements and press releases about the cost-effective wonders of SORNA. Perhaps these results will not be misreported by the media and bloggers.