Bias in how predicted probabilities are used
As mentioned in the introduction to this section, bias may also be introduced after modeling. That is, steps that decision-makers take to translate predicted probabilities into action can also introduce bias.
Bias introduced when selecting thresholds. As discussed in the previous section (Evaluating and comparing learner performance), predicted probabilities are often converted into predicted classifications in order to prioritize units for intervention. Even if the initial, predicted probabilities are unbiased, bias may be introduced when a threshold is selected. In particular, bias can occur if the distribution of risk scores within the resulting groups differs by race. For example, if detention is recommended for pretrial risk scores above a specified threshold, there could be bias if the proportion of those above the threshold is different for Black defendants than it is for White defendants.
Bias introduced by relying on factors that override risk-assessment scores. Guidelines for pretrial detention decisions typically incorporate information other than results from a risk assessment (for example, some jurisdictions may automatically “bump up” the effective risk score of a defendant if the arrest involves a violent crime). Bias may be introduced if this other information is both not additionally predictive of an outcome (beyond the baseline data used) and occurs differentially across groups. For example, if a particular targeted charge is more common among Black defendants than among White defendants, but that charge does not predict failure to appear beyond the baseline prediction, then a “bump-up” based on that charge may introduce bias that does not exist in the underlying risk-assessment algorithm. A similar issue could readily occur in other contexts as well if decision-makers combine other information with predicted analytics results to guide recommendations about differential services.
Bias in implementation. Finally, bias can be introduced in terms of how predictive analytics results are used in decision-making. For example, one source of bias in the pretrial process is differential decision-making by judges (and others who make detention decisions): Judges (or other stakeholders) may impose harsher release conditions on one group than another, even when they receive the same release recommendation for similar cases and contexts. One purpose of predictive analytics is to ameliorate such biases, but this potential for bias in implementation of predictive analytics underscores the need to assess the bias of the fully implemented system as well as predictive tools themselves.