Lecture 18: Regression:

Question 1

What type of variables are seen in regression?

Answer 1

Regression: Continuous dependent and independent variables

Question 2

What’s the difference between QUALITATIVELY and QUANTITAVELY?

Answer 2

Qualitative information describes qualities, characteristics, or experiences using words.

Quantitative information measures, counts, or quantifies using numbers and statistics.

Question 3

What does a simple linear regression describe?

Answer 3

Simple linear regression describes the linear relationship between a predictor variable, plotted on the x-axis (distance from East Africa) , and a response variable, plotted on the y-axis (genetic diversity).

Question 4

What variable regresses on the other?

Answer 4

We say “regress Y on X”

Ex: “regress genetic diversity on distance from Africa”

Question 5

What is the observed value?

Answer 5

The dots on the scatter plot

Question 6

What is the line that follows the general trend of the scatter plot called the fitted regression line?

Answer 6

Fitted regression line representing predicted values for any given value of X (proportion black). Best fit of the data → aid in making predictions in the data

Question 7

What does the fitted regression line aid in?

Answer 7

aid in making predictions in the data

Question 8

What is the predicted value?

Answer 8

Values along the fitted regression line

Question 9

What is the Residual value e?

Answer 9

• The difference (deviation) between the observed and predicted values.
• Can have positive and negative residuals!

Question 10

What algorithm does a regression model use and what does it assure?

Answer 10

A regression model uses an algorithm called “ordinary least squares (OLS)” that assures that the residual (deviation) values are as small as possible given the data. In other words, OLS maximizes the predicted values to be as closest as possible (in average) to the predicted values.

Question 11

The regression line through a scatter of points is described by the following equation:

Answer 11

Y = a + bx

1) 𝒀 is referred as response variable (or also dependent variable).
2) 𝑿 is referred as explanatory variable
3) a = intercept: The predicted value of Y when X is zero (unit is the same as in Y)
4) b = slope: the rate of change in y as x changes

Question 12

Why must you be careful when interpreting the intercept of a regression line?

Answer 12

A meaningful interpretation is only possible if X can truly be zero AND if the data include values close to zero (not the case here) → this is an issue.

Question 13

What is the unit of the intercept of the regression line?

Answer 13

The unit attached to the intercept is the same as the response variable (i.e., years).

Question 14

Why is the intercept is useful for prediction?

Answer 14

Because it represents the addition (offset) required to correctly position the regression line so that predictions match the observed data.

–> Different intercepts would lead to predicted values that are either too high or too low.

Question 15

Define the slope

Answer 15

Because X is expressed in proportions (i.e., 0 to 1), then the slope is the increase of the response variable (age) when the predictor increases 100%, i.e., when X = 1.

Question 16

What does Ŷ (y hat) stand for?

Answer 16

Predicted values on the regression line

Question 17

What are residuals 𝜀?

Answer 17

Residual values 𝜀 are the difference (deviation) between the observed and predicted values

Question 18

True or false: Each observation in the data has a predicted & residual value

Answer 18

TRUE

Question 19

What is the purpose of the OLS or ordinary least squares?

Answer 19

Trying to minimize the sum of the squares of the residuals

Question 20

What is the aim of a regression line?

Answer 20

A regression model aims at predicting the average Y based on X, i.e., predict the average male lion based on their proportion of black spots

Question 21

What does the line of best fit for a regression line minimize?

Answer 21

The line of best fit minimizes the average distance between data and fitted line, i.e., the residuals.

Question 22

How do we find the best line of a regression?

Answer 22

To find the best line, we must minimise the sum of the squares of the residuals

Question 23

Why does the model to minimise the sum of the squares of the residuals to find the line of best fit use squares instead of square root?

Answer 23

Use squares such that it becomes a variance equation and can use the F-distribution

Question 24

What is the H0 and HA of a t-test in statistical hypothesis testing of a regression model?

Answer 24

1) H0: the statistical population slope 𝛽 = 0 (i.e., Y can’t be predicted by X).
2) HA: the population slope 𝛽 ≠ 0 (i.e., Y can be predicted by X).

Question 25

When is regression significant?

Answer 25

Regression is significant when the slope is different from zero

Question 26

As with any estimate based on sample data, slopes can differ from zero even when the true population slope is zero, simply due to _____

Answer 26

sampling variation.

Question 27

How is regression tested using something similar to a one sample t-test?

Answer 27

The regression slope b divided by its standard error can be used to test the null hypothesis that 𝛽 = 0. This is similar to the one-sample t-test: --> Dividing data by the standard error becomes t-distributed

Question 28

If P < 0.05 for a regression, what does this mean?

Answer 28

reject the H0 and conclude that the regression model can predict Y values.

Question 29

What occurs when there's an increase in residuals?

Answer 29

Increase in residuals have more variation leading to greater standard error and a smaller t-value such that there’s less statistical power to reject the null hypothesis

Question 30

What does statistical testing of regression depend on?

Answer 30

Both the slope and the residual variation

Question 31

What is measured using the quantity called the “coefficient of determination” or the “famous” R^2?

Answer 31

We can measure the fraction of variation in Y (age) that is “explained” by X in the estimated linear regression model using a quantity called “coefficient of determination” or the “famous” R2:

Question 32

What is the formula for R^2?

Answer 32

Sum of the squares of regression/Sum of the total

Question 33

What is the Sum of the total in R^2?

Answer 33

The maximum amount of variation in Y that could be explained by any linear regression model is the total sum-of-squares of Y

Question 34

What is the sum of the squares of regression in R^2 and what is it a measure of?

Answer 34

The amount of variation in Y (age) that the regression model with proportion of X (black spots) as a predictor is the regression sum-of-squares The predicted values - the mean of the observed values → a measure of quality.

Question 35

What does it mean when R^2 is = to 0.6238 for example?

Answer 35

- We state then that the regression model explains 62.38% of the total variation in Y. Ex: 62% of the age of lions can be predicted using the proportion of black spots of the noses of male lions.

Question 36

Do regression of Y on x always imply dependency?

Answer 36

NO: SPURIOUS CORRELATION: --> correlation between 2 variables having no causal relation.

Question 37

Define SPURIOUS CORRELATION

Answer 37

Correlation between 2 variables having no causal relation.

Question 38

What are confidence bands for regression lines?

Answer 38

Confidence bands describe the uncertainty around the mean relationship between X and Y.

Question 39

What does it mean when we are 95% confident that the slope will fit within this interval for a regression line?

Answer 39

Because individual data points include residual variation, many observed values will naturally fall outside the confidence bands. We are interested in the confidence of the slope

Question 40

What is the prediction interval for a regression line?

Answer 40

Prediction bands describe the uncertainty around individual observations, so they are always wider because they include both uncertainty in the mean and residual variation.

Question 41

Confidence versus prediction intervals:

Answer 41

1) Confidence bands describe the uncertainty around the mean relationship between X and Y. 2) Prediction bands describe the uncertainty around individual observations, so they are always wider because they include both uncertainty in the mean and residual variation.

Question 42

What are the issues involving extrapolation in regression lines: predicting Y for X-values beyond the range of the data:

Answer 42

- Ear length= 55.9+0.22(age). Our ears grow longer, about 0.22mm per year. - The intercept here predicts ear length at birth (X=0 years); a baby does not have ears of 56mm (i.e., 5.6cm)!! - Slope is changing and at birth, the predictions do not hold as babies do not have ears of 5.6cm --> So, predictions hold well within the range of X values but not outside.

Question 43

Define extrapolation?

Answer 43

Extrapolation: the action of estimating or concluding something by assuming that existing trends will continue or a current method will remain applicable.

Question 44

What must you ensure about the predictor value of a regression line?

Answer 44

Ensure that the distribution of predictor value is approximately uniform within the sampled range: the standard error cannot tell you that: - Uniform distribution of the predictor variable must occur or else it indicates biased sampling leading to a Type I error

Question 45

What are the 6 assumptions of regression models?

Answer 45

1) Linearity; It is critical to graph the data 2) All observations have similar influences on the regression model 3) Residual variation is normally distributed; that is where the confidence interval comes from 4) Residual variation is homoscedastic (constant across the range of X values) 5) Values of X (predictor) is measured without error (hard to assess, often assumed) 6) Residuals are independent: this is the assumption in which data are sampled randomly

Question 46

What is the linearity assumption of the regression model?

Answer 46

- Plotting the residuals against predictor values is critical in assessing whether a linear model is appropriate. - The horizontal line is the average of residuals (which is always zero as a result of the fitting method). - If variance is greater in different parts of the line, this indicates lack of linearity or heteroscedasticity --> dots should be mostly clustered around the line

Question 47

What is the assumption that all observations must have similar influences on the regression model?

Answer 47

Francis Anscombe’s quartets: Quartet 1 is the only appropriate in the sense that all observations have the same influence on the model, i.e., removal of one observation won’t affect the model much. There are different methods to estimate the influence of each observation on the model (advanced level).

Question 48

What are Francis Anscombe’s quartets?

Answer 48

It comprises four data sets that have nearly identical simple descriptive statistics and regression models. Yet, they have very different distributions and appear very different when graphed. These data demonstrate both the importance of graphing data before analyzing it and the influence of influential observations (outliers). --> All Quartets have the same regression model and R^2

Question 49

Which quartet is appropriate for a regression model?

Answer 49

Quartet 1: Data is scattered but follows a similar trend

Question 50

What is the assumption that residual variation is normally distributed (that is where the confidence interval comes from) for the regression model?

Answer 50

- Normality assumption: At each value of X, there is a normally distributed population of Y-values with the mean on the true regression line. - One can estimate the model even if residuals are not normally distributed, but one cannot generalize the model to predict other observations in the statistical population or make inferences (e.g., p-value, confidence intervals, t-tests, ANOVAs).

Question 51

What is the assumption that residual variation is homoscedastic (constant across the range of X values) in regression models?

Answer 51

- Homoscedasticity assumption: At each value of X, there is a normally distributed population of Y-values with the mean on the true regression line. The variance of the Y-values is assumed to be the same for every value of X. - Can't generalize the model to predict other observations if this assumption isn't met

Question 52

What is the assumption that values of X (predictor) is measured without error (hard to assess, often assumed) for a regression model?

Answer 52

When values of x are measured with error, it results in a change in slope that does not properly predict the values of Y. --> ERROR IN X REDUCES SLOPES.

Question 53

What is an approach to the problem of measuring x values with error in regression models?

Answer 53

One approach to this problem is the so-called Type II regression models: 1) Vertical: Residuals for Type I regression Error in Y but not in X 2) Perpendicular: Residuals for Type II regression Error in both Y and X

Question 54

When performing a boxplot of both Type I and Type II regression, what can be seen?

Answer 54

1) Type I: values more clustered around the mean with smaller whiskers indicating low variability in data 2) Type II: wider boxpot so values are less clustered around the mean and longer whiskers indicating more variability in the data

Question 55

Type II regression is not biased but greater standard error (sampling variation): but why?

Answer 55

This is obvious because both X and Y have errors.

Question 56

What must one be careful of when residuals are non-independent in regression models?

Answer 56

When residuals are non independent, one should be careful about making inferences (e.g., p-value, confidence intervals, t-tests, ANOVAs)

Question 57

What are 2 tests that can be used to test whether the regression slope differs from zero?

Answer 57

1) Using a t-test 2) Using ANOVA (same H0 and HA)