Hypothesis Testing

Gentle Intro

• Hypothesis
  • after Popper Logic
  • every hypothesis has to be able to be disproven
• null-Hypothesis
  • default value (status quo) for a parameter (until proven false)
  • like defendant in court … unguilty until proven otherwise
  • denoted as $H_0$
• alternative Hypothesis
  • deviation from current knowledge
  • must be proven to be valid
  • denoted as $H_a$

Example Machine

• machine must produce with mean diameter of 0.5 inch
  • $H_0:d=0.5in$
  • $H_a:d\ne 0.5in$

Testing

• possible outcome of any Test
  • reject the null hypothesis
    • finding a non-white swan (significant result)
  • fail to reject the null hypothesis
    • even tho I fail to reject, I still do not accept the null-Hypothesis
      • there can still be a black swan out there
    • in reality, large enough sample size might result in “impractical” status
      • no further research to be done
• rejecting is positive

Errors

• Errors
• $\text{standard error}=\frac{\text{standard deviation}}{\sqrt{n}}$
  • standard deviation … data - how spread out the data points are
  • standard error … meaning - how relevant/meaningful the conclusions are

Ingredients

• confidence level
  • e.g. 99%
• rejection region
  • defining when $H_0$ is rejected in favor of $H_a$
  • e.g. when arbitrary experiment result is greater than 5
  • rejection region is always outside of confidence interval
• test statistic
  • depends on problem we have

Interpretation

• when result is inside confidence interval
  • i.e. outside the rejection region
  • we know that we cannot reject $H_0$, but still not accept it
  • at the current confidence level

Tests

Population Mean

One-Tailed

• upper $H_a:\Theta >{\Theta }_0$ or lower $H_a:\Theta <{\Theta }_0$
• Theta $\Theta$ (measured) and ${\Theta }_0$ (expected) are placeholder for the corresponding values compared
• ignored in this course, but not hard to grasp or adjust the formulas

Two-Tailed

• two-tailed $H_a:\Theta \ne {\Theta }_0$
  • $H_0:\mu ={\mu }_0$
  • $H_a:\mu \ne {\mu }_0$
• then we collect sample data and get $\bar{x}$ and $\sigma$
  • or $s$ sample standard deviation if $\sigma$ is not known
• therefore for large samples: $t_{\text{stat}}=\frac{\bar{x}-{\mu }_0}{\frac{\sigma }{\sqrt{n}}}\sim N(0,1)$
  • for small samples: $t_{\text{stat}}=\frac{\bar{x}-{\mu }_0}{\frac{\sigma }{\sqrt{n}}}\sim \text{t-distribution}$
• choose significance level $\alpha$
  • reminder: $\alpha$ = chance of Type I error
  • region within confidence interval → do not reject $H_0$
  • region outside confidence interval → reject $H_0$
  • confidence interval can be constructed without data!
    • only distribution type, sample size and $\alpha$ needed
• $z$-critical value (end points of rejection region)
  • $z_c=qnorm(\frac{1-\alpha }{2})$ if $n$ is large (> 30) → CLT
  • $z_c=qt(\frac{1-\alpha }{2})$ if $n$ is small and population is normally distributed

Population Proportion

• follows Large Sample Confidence Intervals
  • $\hat{p}=\frac{1}{n}\ast {\sum }_{i=1}^n\text{Bernoulli}(p)$
• $\hat{p}\approx N(p,\sqrt{\frac{p(1-p)}{n}})$

$p$-Values

• the probability of obtaining a sample “more extreme” than the one observed in the data set, assuming that $H_0$ is true
• basically reversing the calculation
  • finding $\alpha$ for the given $\bar{x}$ (two-sided CI)
• leaving it up to the reader to interpret the result
• p-value =
  • $2\ast P(\text{observed z}<Z)$ for $\bar{x}<\mu$ → $Z$ will be negative
  • $2\ast P(\text{observed z}>Z)$ for $\bar{x}>\mu$ → $Z$ will be positive