L3 - Least Squares Method

Question 1

Give the Simple Linear Regression function…

Answer 1

Yb(x) = b0 + b1x + error

Question 2

What is the formula for the Y-intercept of the regression line?

Answer 2

b0 = y median - (b1 * x median)

Question 3

What is the formula for the slope of the regression line?

Answer 3

b1 = r * (sy / sx)

Question 4

What is the goal of the simple regression function?

Answer 4

To achieve as small error as possible.

Question 5

What is the formula for the error of the simple linear regression function?

Answer 5

Transpose the simple linear regression function with respect to E

Error = Yb(x) - b0 - b1x

Question 6

What does MSE find?

Answer 6

The average error between the regression line and the data points.

Question 7

Why do we square the errors for MSE?

Answer 7

• We only want to deal with positive values.
• If we summed, the positive and negative values would cancel each other out.

Question 8

What is the advantage of MSE?

Answer 8

• Sensitive to anomalies, thus can reduce their effect.

Question 9

What is the goal of MSE? What must we tune to obtain this?

Answer 9

• Achieve an error as close to 0 as possible.
• We tune b0 and b1 to do this.

Question 10

What is the Least Square Method?

Answer 10

• A method of linear regression in which we tune b0 and b1 in order to obtain the lowest SSE ( Sum of Squared Error )

Question 11

What are the 4 steps to the Least Squares Method?

Answer 11

1. Transpose the system of equations with respect to Error.
2. Square and sum equations. Minimise as much as possible.
3. Calculate partial derivatives with respect to b0 and b1
4. Use elimination to solve the remaining system of equations.
5. Result is b0 and b1 with the smallest error.

Question 12

What do we compare the Least Squares Method against?

Answer 12

The mean of Y, which is the most basic regression line.

Question 13

What are the 2 types of variations encountered in LSM?

Answer 13

Unexplained Variations - The error that can’t be explained by the independent variable.

Explained Variations - The error that can be explained by the independent variable.

Question 14

How do we calculate the Total Variation of LSM?

Answer 14

Total Variation = Unexplained variation + explained variation

Question 15

What is the Sum of Squared Residual (SSR)?

Answer 15

Measures the level of variance in the error of a regression model.

Question 16

What is the Sum of Squared Error (SSE)?

Answer 16

Measures the difference between the predicted values and the error mean (MSE).

Question 17

What is the coefficient of determination?

Answer 17

Value between 0 and 1 that informs us of the prediction quality of the model.

1 is better.

Question 18

What is the formula for R^2?

Answer 18

R^2 = 1 - (SSE/TSS)

Question 19

In regression formulas, what is omega? When do we care about it?

Answer 19

• Omega is the regression coefficients.
• We care when we are establishing feature importance (interpreting model over prediction).

Question 20

What is the difference between Prediction and Interpretation?

Answer 20

Prediction: Using model to predict a value. We focus on performance metrics, not omega. Be careful about model becoming black box.

Interpretation: Using model to gain insight into data. We focus on parameters toe stablish feature impact.