Linear Regression

Simple Linear Regression

• question: “is the price of a house dependent on the size ($m^2$)?”

Deterministic vs Probabilistic

• deterministic … exact relationship $y=a+bx$
  • very unrealistic, we are not in a deterministic world
• probabilistic … deterministic component + random error $y=a+bx+\epsilon$
  • oftentimes used
  • when modelling we need to take error into account
  • error term $\epsilon$ also takes care of the data points not taken into account
    • sample vs population

Theory

Info

line of means $E[y]={\beta }_0+{\beta }_{1}x$

• y does not increase, it is expected to increase
• ${\beta }_0$ and ${\beta }_1$ are unknown, we have to find/compute them
  • we want to find the best fitting parameters $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$
• best … least error (distance) between the actual values and the expected values
  • $u_i=y_i-\overline{y_i}$ … error of any data point
  • $SSE=RSS=\sum _{i=1}^n{u}^2=\sum _{i=1}^n(y_i-(\widehat{{\beta }_0}+\widehat{{\beta }_1}{x}_i){)}^2$
    • $RSS$ … residual sum of squares
    • $SSE$ …
• minimize $SSE$ to reach $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$
  • take partial derivatives in respect to $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$
  • solves beautifully, allegedly
    • 2 equations with 2 unknowns
  • not exam relevant, the minimizing function(s) will be given on the formula sheet, just plug in
• method of least squares
  • $\widehat{{\beta }_1}=\frac{S_{XY}}{S_{XX}}=r_{XY}\ast \frac{\sqrt{S_{XY}}}{\sqrt{S_{XX}}}$
  • $\widehat{{\beta }_0}=\bar{y}-\widehat{{\beta }_1}\bar{x}$
• coefficient of determination
  • $R^2=r_{XY}^2$
    • why…? long formula
  • $R^2=\frac{\sum _{i=1}^n(y_i-\bar{y}{)}^2-\sum _{i=1}^n(y_i-\hat{y}{)}^2}{\sum _{i=1}^n(y_i-\bar{y}{)}^2}$
  • relation of variation in X and variation of the error
    • variation of the error ⇒ variation of X
  • relation of variation in X and variation in variation in X
  • $R^2$ is always between 0 and 1

Multiple Regression Model

• question: “which parameters is the price of a house dependent on?”
  • e.g. size, # of bedrooms, distance to nearest grocery/school/public transport, has garden
• similar, just more variables
  • again method of least squares (OLS estimation)
  • F-Distribution test

Important

with R … stars in p-values are significant keep them in your model, the more stars the merrier

• multiple $R^2$ takes also number of variables into account
  • when comparing models with different number of variables, look at adjusted $R^2$ values