The Theory: Understanding Linear Regression
Linear Regression is a statistical method used to model the linear relationship between a dependent variable (Y) and one or more independent variables (X). The goal is to create an equation that minimizes the sum of squared residuals (the difference between predicted and actual Y values).
- Simple Linear Regression (SLR): Uses one independent variable (X) to predict the dependent variable (Y). The equation is Ŷ = b₀ + b₁X, where b₀ is the intercept and b₁ is the slope (coefficient).
- Multiple Linear Regression (MLR): Uses two or more independent variables (X₁, X₂, ...) to predict the dependent variable (Y). The equation is Ŷ = b₀ + b₁X₁ + b₂X₂ + ...
- Pearson Correlation Coefficient (r): Measures the strength and direction of the linear association between two variables. Values range from -1 (perfect negative correlation) to +1 (perfect positive correlation).
- Coefficient of Determination (R²): Represents the proportion of the variance in the dependent variable (Y) that is predictable from the independent variable(s) (X). For Simple Linear Regression, R² = r².
Part 1: Simple Linear Regression and Correlation
Use the data below, which examines the relationship between **Hours Studied per Week (X)** and **Exam Score (Y)** for 15 students. You will need to input this data into the calculator to solve many of the questions.
| Hours Studied (X) | Exam Score (%) (Y) |
|---|---|
| 5 | 65 |
| 7 | 70 |
| 8 | 75 |
| 10 | 78 |
| 12 | 80 |
| 12 | 85 |
| 14 | 88 |
| 15 | 90 |
| 16 | 91 |
| 18 | 94 |
| 20 | 95 |
| 5 | 68 |
| 9 | 77 |
| 11 | 82 |
| 17 | 92 |
Question 1: Correlation Interpretation
Based on the data, what is the value of the Pearson correlation coefficient (r), and what does its sign and magnitude imply about the relationship?
Hint: r tells you the strength and direction of the linear relationship. Is it close to 1, -1, or 0?
Question 2: Coefficient Interpretation
Assume the calculated regression equation is: Ŷ = 60.5 + 1.8X. Interpret the meaning of the slope coefficient, b1 = 1.8, in the context of study time and exam score.
Hint: The slope explains the expected change in Y for every one-unit increase in X.
Question 3: Making a Prediction
Using the equation from Question 2 (Ŷ = 60.5 + 1.8X), what is the predicted exam score for a student who studies 13 hours per week?
Hint: Substitute X = 13 into the regression equation.
Question 4: Coefficient of Determination (R²)
If the R² for the model created with the provided data is approximately 0.91, what percentage of the variability in Exam Score is explained by the variability in Hours Studied?
Hint: R² is directly interpreted as the proportion of variance explained, expressed as a percentage.
Question 5: Assumption Check
Which crucial assumption of Simple Linear Regression is assessed by examining a scatter plot of the residuals (errors) versus the predicted values (Ŷ)?
Hint: This check looks for a systematic pattern in the errors, which relates to the assumption of homoscedasticity.
Solve with the Simple Linear Regression Calculator
Open Calculator in New TabPart 2: Multiple Linear Regression
Use the data below, which models House Price (Y, in $1000s) as a function of Square Footage (X1, in $100s) and Distance to City Center (X2, in miles).
| Price ($1000s) (Y) | Sq. Ft. ($100s) (X1) | Distance (miles) (X2) |
|---|---|---|
| 350 | 25 | 10 |
| 450 | 30 | 5 |
| 280 | 20 | 15 |
| 520 | 35 | 4 |
| 400 | 28 | 8 |
| 600 | 40 | 3 |
| 310 | 22 | 12 |
| 480 | 32 | 6 |
| 330 | 24 | 11 |
| 550 | 38 | 5 |
| 250 | 18 | 18 |
| 420 | 29 | 7 |
| 500 | 33 | 5 |
| 370 | 26 | 9 |
| 630 | 42 | 2 |
Question 6: Interpreting Multiple Coefficients
Suppose the Multiple Linear Regression model yields the following equation: Ŷ = 100 + 10X1 - 5X2. Interpret the coefficient for distance, b2 = -5.
Hint: In MLR, a coefficient is interpreted as the change in Y for a one-unit change in X while holding all other variables constant (ceteris paribus).
Question 7: Prediction with Multiple Variables
Using the equation from Question 6 (Ŷ = 100 + 10X1 - 5X2), what is the predicted price (in $1000s) for a house with 3000 sq. ft. (X1 = 30) that is 8 miles from the city center (X2 = 8)?
Hint: Substitute both X1 and X2 values into the equation to find Ŷ.
Question 8: Adjusted R² vs. R²
Why do statisticians often prefer to use Adjusted R² over the standard R² when building a Multiple Linear Regression model with many predictors?
Hint: Standard R² has a tendency to increase unrealistically when more variables (even irrelevant ones) are added to the model.
Question 9: Multicollinearity
What problem arises in Multiple Linear Regression when two independent variables, like X1 (Size of House) and X3 (Number of Bedrooms), are very highly correlated with each other?
Hint: This term describes excessive inter-correlation among the predictor variables, making individual coefficient estimates unstable.
Question 10: Null Hypothesis Test
A researcher performs an F-test on the overall significance of their Multiple Linear Regression model. State the null hypothesis (H₀) for this test.
Hint: The F-test assesses if the model, as a whole, is a better predictor than using only the mean of Y. H₀ is always the statement of no effect.