What are the limitations of PA for improving services?

While PA has the potential to provide valuable insights for improving services, it also has many limitations that data scientists need to consider when scoping, implementing and communicating plans and results. The following provides a brief overview of some limitations of PA. We will return to these in more depth later.

  1. Predictions are only as good as the data on which they are built. If the dataset available for analysis is incomplete, inaccurate, or biased, it can lead to unreliable predictions.
  2. There can be a trade-off between model performance and transparency and simplicity. Sometimes the best-performing models are complex machine learning algorithms and can therefore be difficult to explain and can be challenging to deploy and maintain. A lack of transparency can be especially problematic in certain domains with more ethical considerations.
  3. External factors may change: Predictive models rely on patterns in the past to make predictions about the future. But models may not account for unexpected events, external influences, or sudden changes in the environment that can substantially impact the outcome being predicted. As the time horizon extends further into the future, models may become unreliable. For this reason, predictive models need regular monitoring and may require frequent updating.
  4. Predictive analytics tells us nothing about causal relationships. Nor does it tell us that those with the lowest or highest probability of an outcome are the most likely to benefit from a new or revised service or intervention. This evaluation is determined by other analyses (e.g. rigorous evaluations), or is sometimes assumed based on the expertise of those using predictive analytics findings for decision-making.
  5. Ethical Concerns: Predictive analytics can raise many ethical concerns. Ethics is a very important topic, which we turn to next and will return to throughout the course.

Many of the lessons we learn in inferential statistics do not apply to predictive analytics. You can find a short discussion of some of the differences between predictive analytics and inferential statistics here. This is OPTIONAL.

And for fun, Baba Brinkman covers some of these issues in his rap comparing and contrasting data science and statistics. (Click link below.) (Also OPTIONAL)

Back to top