11 P-Values

P-values are a convenient way to communicate the results of a hypothesis test. When communicating a P-value, the reader can perform the test at whatever Type I error rate that they would like. Just compare the P-value to the desired Type I error rate and if the P-value is smaller, reject the null hypothesis.

Formally, the P-value is the probability of getting data as or more extreme than the observed data in favor of the alternative. The probability calculation is done assuming that the null is true. In other words if we get a very large T statistic the P-value answers the question “How likely would it be to get a statistic this large or larger if the null was actually true?”. If the answer to that question is “very unlikely”, in other words the P-value is very small, then it sheds doubt on the null being true, since you actually observed a statistic that extreme.

  • Most common measure of statistical significance
  • Their ubiquity, along with concern over their implementation and use makes them controversial among statisticians
    • Statistical : A likelihood paradigm by Richard Evidence
    • The hilariously titled; The Earth is Round (p < 0.05) by Cohen
  • Some positive comments:
    • Simply Statistics
    • Normal deviate
    • Error Statistics

11.1 What is a P Value?

Idea: Suppose nothing is going on, how unusual is it to see the estimate that we got?

Approach:

  1. Define the hypothetical distribution of a data summary (statistic) when “nothing is going on” (null hypothesis)
  2. Calculate the summary/statistic with the data we have (test statistic)
  3. Compare what we calculated to our hypothetical distribution and see if the value is “extreme” (p-value)

The P value is the probability under the null hypothesis, of obtaining evidence as extreme or more extreme than that obtained.

If the p value is small, then either \(H_0\) is true and we have observed a rare event, or \(H_0\) is false.

Suppose that you get a \(T\) statistic of 2.5 for a 15sf testing \(H_0:\mu = \mu_0\) versus \(H_0 : \mu > \mu_0\). What’s the probability of getting a \(T\) statistic as large as 2.5?

pt(2.5, 15, lower.tail = FALSE) = 0.01225

11.2 The Attained Significance Level

Our test statistic was 2 for \(H_0:\mu_0=30\) versus \(H_a:\mu>30\). Notice that we rejected the one sided test when \(\alpha = 0.05\), would we reject if \(\alpha = 0.01\), how about \(0.001\)?

11.3 Example of Your Friend Having Children

that your friend has 8 children and 7 of them are girls, none are twins. If each gender has an independent 50% probability for each birth, what’s the probability of getting 7 or more girls out of the 8 births?

\[H_0: p=0.5~~~H_a:p>0.5\]

To get the two sided p-value for this case, calculate the tail for 7 and above and calculate the tail for 7 and below, take the smaller one and double it.

## [1] 0.03515625
## [1] 0.03515625

11.4 Poisson Example

  • Suppose that a hospital has an infection rate of 10 infections per 100 people / day at risk (rate of 0.1) during the last monitoring period.
  • Assume that an infection rate of 0.05 is an important benchmark.
  • Given the model, could the observed rate being larger than 0.05 be attributed to chance?
  • Under \(H_0: \lambda=0.05\) so that \(\lambda_0 100= 5\)
  • Consider \(H_a: \lambda>0.05\)

Keep in mind that the Poisson Rate is \(0.05\times100\) person days at risk, which gives us \(5\). We want to calculate the

To solve this problem, we want to use the ppois() function:

## [1] 0.03182806

This result tells us the probability of obtaining 10 or more infections, if the true rate of infections we should have seen (for 100 person days at risk) is 5.