Predictive Analytics for Improving Services
Overview
Defining PA for improving services
Scoping a PA project
Conducting a PA proof-of-concept
Data for PA: PART 1
- Generalizing to new data
- Data workflow
- Identifying data
- Designating data for testing
- Training and validating learners
- Additional considerations
Data for PA: PART 2
Introduction to the PA Toolset
- Overview
- How to access toolset
- How to specify scope in toolset
- How to specify learners in toolset
Modeling approaches
- Regression for predicting binary outcomes
- Machine learning (ML) algorithms for predicting binary outcomes
Evaluating and comparing predictive performance
- Prediction output from learner training
- Metrics for evaluating predictive performance
- Visualiziations of predictive performance
- Guidance for selecting metrics
Evaluating and comparing bias
- Sources of bias
- Metrics for assessing bias
- Visualizations for assessing bias
- Guidance for selecting the metrics
- Addressing bias
Selecting and validating a learner
- Considerations for selecting a learner
- Validating a learner
Understanding and communicating results
- Results at the individual level
- Results at the aggregate level
- What can we say about which predictors matter most?
- Reviewing limitations and ethical considerations
Deploying predictive analytics
- Deciding whether PA should be deployed
- Approaches for deployment
- Monitoring performance and fairness over time
Additional optional material
- Predictive Analytics vs. Inferential Statistics

Defining learners

Here we present a framework for conducting a PA proof-of-concept. The framework focuses on specifying and comparing “learners.”

A learner is a combination of:

A predictor set: a set of measures that are available at the prediction timepoint and that are potentially predictive of the outcome of interest.
A modeling approach: a method for combining and weighting measures, e.g., regression or a particular machine learning algorithm.

Machine learning algorithms automate model building using a series of steps that are driven by patterns in the data - rather than relying on functional forms specified by an analyst. Examples include decision trees, random forests, stepwise regression, and support vector machines, among many others.