39 Types of Errors

39.1 Basic Terms

  • False positives, False negatives, True positives and True negatives. See Sensitivity and Specificity.

  • Sensitivity = \(Pr(Positive ~Test~|~Disease)\)

    • When you’re sick, what’s the probability that we predict that? \[\frac{TP}{TP+FN}\]

  • Specificity = \(Pr(Negative ~Test~|~ No Disease)\)

    • If you’re actually healthy, what’s the probability that we predict that? \[\frac{TP}{FP+TN}\]

  • Positive Predictive Value = \(Pr(~Disease~|~Positive ~test~)\)

    • What’s the fraction of people who we call sick, who are actually sick? \[\frac{TP}{TP+FP}\]

  • Negative Predictive Value = \(Pr(~Disease~|~Negative ~test~)\)

    • What’s the fraction of people who we call healthy who are actually healthy? \[\frac{TN}{TN+FN}\]

  • Accuracy = \(Pr(~Correct ~Outcome)\) \[\frac{TP+TN}{TP+FP+FN+TN}\]

  • Using a confusion matrix to diagnose your model, Accross the top is what you actually are and along the side is what your model has predicted you are. Using the breast cancer dataset for example, the top left corner gives you your true positives and the top right corner gives you your false positives.

39.2 Screening Tests

Assume that n = 10,000 and that some disease has a 0.1% prevalence in the population. Assume we have a test kit for that disease that works 99% sensitivity and 99% specificity. What is the probability of a person having the disease given the test result is positive, if we randomly select a subject from:

  • The general population?
  • The high risk sub-population with 10% disease prevalence?
so if we made a confusion matrix for this problem, it would look something like this:

39.3 Common Error Measures For Continuous Data

Mean Squared Error (MSE) \[\frac{1}{n} \sum_{i=1}^n (Prediction_i ~ Truth_i)^2\]

Root Mean Squared Error (RMSE) \[\sqrt{\frac{1}{n} \sum_{i=1}^n (Prediction_i ~ Truth_i)^2}\]

  1. Mean squared error - Continuous data, sensitive to outliers

  2. Median absolute deviation - Continuous data, often more robust

  3. Sensitivity (recall) - If you want few missed positives

  4. Specificity - If you want few negatives called positives

  5. Accuracy - Weighs false positives / negatives equally

  6. Concordance - One example is Kappa (Multiclass data)