Windypundit

As I explained in the first post of this series, I’m trying to build a computer model of certain aspects of the criminal justice system. This week, it’s not going well.

My goal is to build a model of how our justice system handles plea bargains, with an eye toward understanding (1) why a system built around the highly formalized decision-making mechanism of the trial doesn’t actually have many trials, and (2) what we can do about it. As I am neither a professional systems modeler nor a lawyer, this is mostly intended as a learning experience.

I plan to eventually give the model a bunch of parameters such as time spent waiting for trial, case quality, sentencing regime, defendant risk tolerance. If I do a good job building the model, then I should be able to find a set of parameters that make the model produce results that look like our criminal justice system: Lots of plea bargains, very few trials, etc. Then I can experiment with ideas for reform by tweaking the model parameters to see how the model responds.

I’m taking an iterative approach to building the model, starting with a skeleton that has some of the major moving parts of the criminal legal process, and then elaborating on parts of the model to flesh it out into something more realistic.

The most recent model is very simple: There’s only one crime, people are arrested at a fixed rate, they wait in jail for a fixed time, and then they have a trial, after which they may go to jail for a fixed period. I suspect my lawyer readers will be annoyed to learn that the entire trial — the core of their profession — is modeled as a single draw from a random distribution to get the verdict. Plea bargaining, my whole reason for doing this, isn’t even in the model yet.

It’s that last problem that I tried to fix. I’m trying to add a plea bargaining phase somewhere between arrest and trial, and I’d like to tell you about it. This story will not end well.

I started by thinking about the basic decision the defendant has to make during plea bargaining: Take the prosecution’s offer or reject it and go to trial. I think there are two main question to address: What does the defendant know? And what does the defendant prefer?

I’m assuming the defendant has knowledge of a few basic things:

• The amount of time in the plea offer from the prosecution.
• The possible outcomes of the trial, consisting of:
  • The probability of being found guilty.
  • The sentence they will serve if found guilty.

Here you see more of my assumptions: Everybody has the same information, and it’s correct information as far as it goes. Note that I’m not saying anyone knows how the trial will turn out ahead of time — I’m modeling it as a draw from a random variable — but I am assuming that both the prosecutor and defense know what that probability is, and that both can use it to make decisions.

When thinking about a random event, it’s helpful to consider the expected outcome, which is calculated as the sum of the value of each outcome times the probability of that outcome. If you buy a raffle ticket which has a 1-in-10 chance of winning $50, we can calculate the expected value of your ticket as follows: Your 1-in-10 chance is a way of saying p = 0.1 for winning, so we multiply that by the payoff, $50, and get $5. Your 9-in-10 chance of losing has p = 0.9, but you get nothing for that, so the value of losing goes to $0. Add up the expected value for all possible outcomes (5 + 0), and you get the expected value of your ticket: $5.

Similarly, if the defendant knows he will be found guilty half the time (p = 0.5), and he knows the sentence is 10 years, he can calculate the expected sentence as (0.5 x 10 =) 5 years. Once he receives the prosecutor’s plea offer, he knows everything he needs to know (under this model) to make a decision.

The key issue controlling the defendant’s decision is then his attitude toward the risk of going to trial. If the prosecutor’s plea offer is 5 years in jail — exactly equal to the expected sentence calculated above, then you could argue that the defendant should be indifferent to the choice between taking the offer or going to trial, since the expected sentence is 5 years either way.

The problem with that approach is that it ignores risk, and people don’t usually ignore risk. All other things being equal, people usually prefer to reduce their risk. There is a massive amount of evidence for this, including the existence of the entire insurance industry, where customers pay fixed premiums to avoid the risk of having to suffer losses. If this risk aversion extends to plea bargaining decisions, a defendant will choose a certain sentence of 5 years in jail over a 50% chance of spending 10 years in jail, even thought the latter also includes a 50% chance of going free.

But if the defendant prefers a 5-year plea over a trial with a 50% risk of a 10-year sentence, what happens if the prosecutor makes an offer of 5 years and 1 month instead? Well, the insurance industry is pretty profitable, which means that people are paying more in premiums than their insurance companies are paying out to cover losses, which means that (on average) customers lose money when they buy insurance. So I think it’s very likely that a defendant will agree to lose an extra month of life in prison to protect against the risk of losing an extra five years of life in prison. If criminal defendants are risk averse, they would be willing to accept a plea bargain for more time than the mathematically expected sentence.

I am modeling this risk aversion by calculating an adjustment to the probability of guilt that represented how that probability would feel to a risk-averse defendant. E.g. a 50% (p = 0.5) probability of guilt with a 10-year sentence has a mathematically expected sentence of 5 years, but to a risk averse defendant, even a 6-year sentence might be preferable to a 50% risk of a 10 year sentence.

I decided to calculated the adjusted risk-averse factor by calculating an expected probability of an acquittal as 1 – p, then raising it to a power specified as a model parameter, and then converting back to a probability of guilt by subtracting it from 1 again. E.g.

1 – (1 – p)^a

where

p is probability of guilt

a is the risk aversion model parameter

I picked this method of modeling risk aversion because (1) I vaguely remember reading an article about risk aversion that modeled it like this, (2) it has the nice property that p = 0 and p = 1 both map to themselves, so it stays in range, and (3) I like the look of the curve. Here, for example, are the curves with a set to 1, 1.5, and 2 for comparison. Remember a = 1 means no risk aversion.

Now that I had a simple model for defendant choice, it was time to model the prosecution.

One of the fundamental differences between defendants and prosecutors, at least for purposes of my model, is that defendants are individuals who seek their own advantage in a single criminal case, whereas prosecutors work together across hundreds or thousands of trials to achieve a common organizational goal. So while I feel reasonably comfortable in assuming for purposes of the model that most criminal defendants will try to minimize their risk-adjusted prison sentence, I’m not nearly as confident in figuring out what exactly prosecutors are trying to achieve.

To keep the model simple, I’m going to assume that prosecutors want the opposite of what defendants want. That is, if defendants want to minimize the risk-adjusted sentence they expect, prosecutors want to maximize the aggregate expected prison sentence across all defendants.

You’ll notice I didn’t say the prosecution goal was risk-adjusted. That’s because if defendants are like insurance buyers, prosecutors are like insurance companies. For an individual home owner, the loss of their house to fire would be financially devastating without an insurance policy. But to the company that issued the policy, it’s just business. They issued policies for tens or hundreds of thousands of homes, and they are well aware that a bunch of them are going to burn down every year. Meanwhile, they will be collecting premiums on all of them, and very nearly all of them will survive through the year with no loss whatsoever. In investment terms, insurance companies are “diversifying away” the risk.

Prosecutors can do something similar. Using the example from above, a defendant going to trial against a 50% chance of a 10-year sentence has a mathematically expected sentence of 5 years, but he’s not going to serve 5 years. He’s going to serve either zero or 10 years. On the other hand, a prosecution team that deals with 100 of these trials will have a mathematically expected sentence of 5 years per case, which adds up to a aggregate expected sentence across all cases of 500 prisoner-years. And they’re actually going to get about 500 prisoner-years, which is 50% of the theoretical maximum of 1000 prisoner-years. In other words, in the long run, the prosecutor can earn the expected sentence on average.

Furthermore, defendants can only accept or reject the deal that the prosecutor offers them, but the prosecutor’s office can offer whatever deal they want, including dismissal or no deal, to all of the defendants. This means it’s ultimately the prosecutor who gets to decide which defendants get plea bargains.

However, they have to do that within a resource constraint: There’s a limit on the number of trials the justice system can support in any given time period (at least for the short term). So if 100 cases enter the system, there might only be enough capacity in the system for 30 full trials. For the system to keep functioning, the remaining 70 cases will have to be disposed of with a plea bargain.

But how do they choose which defendants get plea deals and which ones go to trial? Well, consider this batch of defendants:

The first three colums show the probability of being found guilty, the fixed 10-year sentence, and the resulting mathematically expected sentence. The fourth column shows the probability of guilt adjusted for risk aversion (using an exponent of 3/4), and then the plea that the defendant would be willing to accept based on that risk adjustment. Finally, the last column shows the difference between the plea and the mathematically expected sentence. I.e. the premium in prison time that the defendant is willing to pay to reduce the risk of a 10-year sentence.

At this point, the prosecution plea bargaining strategy seemed obvious to me: If the goal is to maximize aggregate prison time, then the prosecution can maximize total prison time by offering plea deals to those defendants willing to accept the most additional prison time beyond beyond the mathematically expected sentence. Thus, if they only have resources for 3 trials for the defendants above, they should dismiss the unwinnable case A, plea out B through H, and go to trial on defendants I, J, and K.

That’s when I realized I had screwed up: If the prosecution is trying to maximize aggregate prison time, and if defendants will agree to more than the mathematically expected sentence because they are risk averse, then why would the prosecution ever go to trial on any case? By the assumptions I had made, they could always get more prison time with a plea offer. And yet…trials happen.

There’s a mistake in here somewhere. Possibly more than one. Either I’ve oversimplified my model, or else it’s flat-out wrong. Or both. A few possibilities spring immediately to mind:

• Maybe I was wrong to assume both sides see the same probability of guilt. The sides in each case could have randomly different views of the quality of any given case. For all I know, all trials occur because the prosecution thinks it will be easier to prove guilt than the defense does.
• Defendants may be risk-preferring instead of risk averse. Conventional wisdom is that criminals are more willing to tolerate risk than law abiding people. Or risk-preferring people, criminal or not, may be more likely to attract police attention.
• Prosecutors may be optimizing for something other than aggregate prison sentence, such as conviction rate. Or prison times may combine in a non-linear manner: E.g. five 20-year sentences may be better for a prosecutor’s career than twenty 5-year sentences.
• Maybe collateral (non-prison) effects of a conviction affect defendants’ decisions in ways that matter. E.g. If any conviction, including by plea bargain, could result in deportation, defendants might grasp at even small chances at trial.
• I may need a more complex model of the trial outcome than just a probability of conviction. Maybe there are important phases or types of cases. Or maybe one side has more information and can consistently make better choices.
• Maybe I need to account for interactions between cases in the prosecutors’ decision making. E.g. taking 5 aggravated battery cases to trial may use up resources that could be used on a murder case, which could affect how they make plea bargains.

Time to go back to the drawing board. At the very least, I’ve got some thinking to do. And maybe it’s time for me to actually read something written by people who study this stuff.

One of the limitations of the Model Justice 1.0 model is that every defendant has the exact same probability of being found guilty at trial. That’s unrealistic, and it’s unrealistic in a way that will matter: The estimation of the probability of a conviction at trial has got to be one of the most important factors in whether a prosecutor decides to offer a plea deal and whether a defendant decides to accept one.

What I want to do in the simulation is to assign a probability p to every defendant who enters the model. This will represent the probability of conviction if the case goes to trial. I.e. if p = 0.25 then there is a 25% chance the defendant will be found guilty by the Verdict action when it runs. If p = 0, there’s no chance of a conviction, and if p = 1, there’s no chance of an acquittal. But before I can do that, I need to answer an important question: How should those probabilities be distributed? Or to put it in the language of the problem domain: How does the quality of the criminal cases vary?

The easiest approach is to assume a uniform distribution. That is, to assume that all possible values of p from 0 to 1 are equally likely, so that a 100-bin histogram of the probability of guilt for 100,000 cases would look something like this, with the values spread out more-or-less equally over all the quality levels.

That doesn’t seem realistic to me. I mean, does it seem right that they would bring just as many cases that will succeed 5% of the time as will succeed 50% of the time? I just don’t believe that the prosecution would bring exactly as many cases at each possible probability of success.

This is where it would be great to use research by professional criminologists, or get advice from people with a lot of experience in the criminal justice system, who might be able to advise me on the quality mix of criminal cases. Alternatively, it would be nice to pull these numbers from a sophisticated model of how the police and prosecutors’ offices build cases. Since I don’t have access to either of those right now, I’m just going to wing it.

I’m going to assume that building a case that will persuade a jury involves trying to achieve a whole bunch of smaller subgoals, and that each of these goals would improve the case in a way that pushes the jury toward a guilty vote — the more goals accomplished, the likely the jury is to convict. If we assume that the prosecutor and police officers have a roughly equal probability of accomplishing each of these goals, then the number of goals accomplished will conform to what is known as a binomial distribution.

A good example of a binomial distribution is the probabilities of various outcomes when we flip ten coins and count how many turn up heads. Since each coin has two possible values, and there are 10 coins, there are 2¹⁰ = 1024 possible outcomes. Of those outcomes, the extreme cases of all heads or all tails can each only occur one way:

HHHHHHHHHH
TTTTTTTTTT

So the probability of either of those cases is 1/1024 ≈ 0.000977.

There are 10 possible ways of having one coin come up heads, one for each coin:

HTTTTTTTTT  THTTTTTTTT  TTHTTTTTTT  TTTHTTTTTT  TTTTHTTTTT
TTTTTHTTTT  TTTTTTHTTT  TTTTTTTHTT  TTTTTTTTHT  TTTTTTTTTH

So the probability of exactly one head in a set of flips is 10/1024 ≈ 0.00977. That’s also the probability of exactly one tail in the set of flips.

Now it gets more complicated. For two coins to come up heads, there are a bunch of ways it can happen. We already know there are 10 ways to get 1 head. So assuming we’ve got one head, there are 9 remaining coins, and any one of those can also come up heads. However, that leads to each combination appearing twice. E.g. we can start with TTTHTTTTTT and then pick anther coin to get TTTHTTTHTT. However, that gives the same result as if we had started with TTTTTTTHTT and picked another coin to get TTTHTTTHTT. In other words, the two possible ways of choosing two heads result in identical sets of flips, so we have to divide by 2. That means the number of ways of getting exactly 2 heads is (10 x 9) / 2 = 45 which occurs with a probabiliy of 45/1024 ≈ 0.0439.

There are 10 x 9 x 8 ways to get to 3 heads, divided by 3 x 2 ways to arrange 3 heads identically, leaving 120 possible ways to get exactly 3 heads or 3 tails, with a probability of 0.117. By similar logic, 4 heads or 4 tails come up 210 ways with a probability of 0.205, and 5 of each comes up 252 ways with a probability of 0.246

If we plot the possibility of each possible number of heads, the probabilities look like this:

For Model Justice 1.1, I’m going to be assuming that the quality of cases charged against defendants assume a similar profile. I don’t know how many factors go into building a criminal case, but I assume it’s a lot more than 10. And since I’m just picking a distribution rather than modeling each of the factors, I don’t really have to care. That means that instead of using a binomial distribution, which depends on a fixed number of factors (you can’t flip 8.7 coins), I’m going to use a continuous approximation to the binomial distribution called the normal distribution, which looks something like this:

This is, I must admit, something of a cop-out. The normal distribution is probably the most common distribution ever, at least when discussing real-world phenomena. (Uniform distributions are common in games and gambling.) In fact, scientists often assume data points are normally distributed even when they really aren’t. It’s one of the sins of statistics, usually committed because there are so many more statistical tools available if you can just assume normality.

However, when dealing with data taken from the real world, there are solid statistical reasons for believing that the normal distribution may be close enough. The real world is often messy, with lots of different effects contributing to the thing you are trying to measure. And when a measurement is influenced by lots of small random variables, each of which can nudge it slightly higher or lower, it turns out that the result often does look a lot like a normal distribution.

This can be illustrated with something called a Galton board. That’s a slanted board with pins sticking up out of it in a regular pattern of offset rows. You drop balls into it at the top, and they trickle down. Each time they hit a pin, they can roll off to the left or right, at which point they will hit a pin from the row below, again rolling off to the left or right, and so on. At the bottom of the board, right below the last row of pins, there’s a set of vertical slots to catch and stack the balls. The height of the stack will tell you how many balls ended up there, and if you dump enough balls into the board, the stacks will follow (more or less) a normal curve.

Here’s a decent video with some explanation (skipping the Galton bio at the beginning):

(You can buy the Galton board used in the video here.)

So that’s my justification for using a normal distribution. I can imagine of a number of reasons why the quality of actual criminal cases wouldn’t follow a normal curve but, at least for now, it’s good enough.

I do need to make some adjustments to fit the model. For one thing, the allowable values for a defendants probability of guilt are from 0 to 1, so I might want to drop the normal curve right in the middle of it, something like this:

There are a couple of problems with that idea.

First of all, the normal distribution is a mathematical construct, so despite the nice appearance here, the left and right tails actually extend out to infinity. So what does it mean if we pick a random value out of the normal curve and the probability of guilt is 3.5? My thinking is that the normal curve doesn’t really represent the probability of guilt. Rather, it’s a crude representation of the quality of the case. So if we get values that are greater than 1, it means the case is more than good enough to guarantee (with p = 1) a conviction. Similarly, a case quality less than 0 means there is no way (p = 0) it could lead to a conviction. So if I get a value outside the range 0 to 1, I’ll snap it to 0 or 1.

The second problem is that there’s no reason to believe the average probability of guilt is exactly 0.5. As I said, the normal distribution actually represents the quality of the case, and it would be quite a coincidence if the average case quality just happened to be exactly good enough to produce a perfect split between guilty and not guilty verdicts. It’s likely that strong cases are more work to put together than week cases, so there will be a tendency to produce a lot of weak cases:

Then again, it’s possible that prosecutors avoid charging extremely weak cases that they don’t think they can win, leaving only the strong cases:

The third problem is that cases may deviate from normality in important ways. For example, the practice of discarding bad cases may lead to a different shape of the left-hand side of the distribution curve. Or maybe there’s always a small chance of a not-guilty verdict, from jury excursions or random black swan events, so that even the best cases can sometimes lose. I have no idea, so I’m going to ignore these possibilities for now. (I may add something in later.)

Incorporating this into the model is pretty straightforward, and you can follow along on the new Model Justice 1.1 model.

The shape of a normal distribution curve is described with two values:

• Mean: The average value. Or the value right in the middle of the curve at the highest point. (One of the handy properties of normal distributions is that the mean, median, and mode are all the same.)
• Standard Deviation: This is basically a measurement of how wide the curve is. The greater the standard deviation, the more the curve stretches out to the left and right.

I’ve replaced the Probability of Guilt slider, and the underlying variable, with Guilt Mean and Guilt Standard Deviation sliders that control the shape of the normal distribution. The values shown will cause this model to produce the same ProbGuilty values as version 1.0:

I also modified the code in the Arrest action to generate a random value from the normal distribution. If the value is outside the range [0,1], it will be snapped to the nearest bound. I.e. Values greater than 1 will be snapped to 1 and negative values will be snapped to 0 so they are valid probabilities. Unfortunately, Insight Maker doesn’t give me any way to show the shape of the generated curve. If you want to get some idea what the curves look like, you can use an online normal curve generator, like this one.

(I’ve also added an ID generator that slaps an ID number on each defendant agent, and I’ve added a global Stats object that collects some internal data that I use for debugging.)

Sadly, after all this discussion, the results from the new model are not noticeably different from the results from the old model. It was generating random guilt probabilities around a specified mean, now the new model is generating random guilt probabilities around a mean that is itself randomly generated around a mean. That tends to look like the same thing.

Nevertheless, these changes are a step toward modeling plea bargaining, because they will keep all defendants from making the same decisions.