What is the Logit function in Logistic Regression?

The Logit function is defined as the natural logarithm of the odds — that is, log(p / (1-p)), where p is the probability of the outcome. It serves as the link function that maps the linear predictor to a probability between 0 and 1.

What is the difference between Linear Regression and Logistic Regression?

Linear Regression predicts a continuous outcome and fits a straight line to the data, assuming normality of residuals. Logistic Regression predicts a probability for a binary (discrete) outcome, fitting an S-shaped sigmoid curve. Linear Regression uses OLS estimation, while Logistic Regression uses Maximum Likelihood Estimation (MLE).

What does an Odds Ratio of 2.5 mean in Logistic Regression?

An Odds Ratio of 2.5 means that for each one-unit increase in the predictor variable, the event is 2.5 times more likely to occur compared to the baseline. Values above 1 indicate increased odds, values below 1 indicate decreased odds.

What does heteroscedasticity mean in regression analysis?

Heteroscedasticity means that the variance of the residuals is not constant across all levels of the independent variable. It typically appears as a fan-shaped pattern in a residual plot, and violates one of the key assumptions of Linear Regression (homoscedasticity).

What is the AUC-ROC curve and why is it important?

The AUC-ROC (Area Under the Receiver Operating Characteristic) curve is a metric used to evaluate classification models such as Logistic Regression. It illustrates the trade-off between sensitivity (true positive rate) and specificity (true negative rate) across all classification thresholds. An AUC of 1.0 indicates a perfect model, while 0.5 indicates no discriminative ability.

What causes multicollinearity and how can you detect it?

Multicollinearity occurs when two or more predictor variables in a regression model are highly correlated with each other. A common sign is a high R-squared value combined with low or insignificant individual t-statistics for the predictors. It can also be detected using the Variance Inflation Factor (VIF), where values above 10 typically indicate a problem.

What is overfitting in regression and how can it be avoided?

Overfitting occurs when a regression model learns the noise in the training data as if it were a meaningful pattern, resulting in poor generalization to new data. It can be avoided by using techniques such as cross-validation, regularization (Ridge or Lasso), or stepwise variable selection to keep the model appropriately simple.

Linear and Logistic Regression Questions and Answers

Q: What is the primary objective of Ordinary Least Squares (OLS) in Linear Regression?

The primary objective of OLS is to minimize the sum of the squares of the vertical deviations (residuals) between the observed values and the values predicted by the regression line. This produces the best-fit line through the data.

Metric	Linear Regression	Logistic Regression
Outcome Nature	Continuous (Scale)	Discrete (Probability/Class)
Relationship Shape	Straight Line	S-Shaped (Sigmoid) Curve
Key Assumption	Normality of Residuals	Independence of Errors

1. What is the primary objective of Ordinary Least Squares (OLS) in Linear Regression?

A. To maximize the likelihood of the sample.
B. To minimize the sum of the squares of the vertical deviations.
C. To reduce the number of independent variables.
D. To ensure the dependent variable is binary.

2. In Logistic Regression, the 'Logit' function is defined as:

A. The square root of the probability.
B. The natural logarithm of the odds.
C. The inverse of the linear predictor.
D. The correlation between X and Y.

3. Which of the following indicates a potential Multicollinearity issue?

A. High R-squared but low individual t-statistics.
B. Low R-squared and high t-statistics.
C. A p-value of 0.01 for the intercept.
D. Randomly distributed residuals.

4. The range of values for a predicted outcome in Logistic Regression is:

A. \((-\infty, +\infty)\)
B. \([0, 1]\)
C. \([-1, 1]\)
D. \((0, +\infty)\)

5. What does an Odds Ratio (OR) of 2.5 imply?

A. The event is 2.5 times less likely to occur.
B. The event is 2.5 times more likely to occur with a unit increase in the predictor.
C. The correlation is 0.25.
D. The slope of the line is 2.5.

6. Which assumption is NOT required for Linear Regression?

A. Linearity.
B. Homoscedasticity.
C. Normality of the independent variables.
D. Independence of errors.

7. The 'Standard Error of the Estimate' measures:

A. The accuracy of the intercept.
B. The dispersion of the observed values around the regression line.
C. The size of the sample.
D. The power of the statistical test.

8. In Multiple Regression, Adjusted R-Squared is used to:

A. Increase the significance of the model.
B. Account for the number of predictors in the model.
C. Remove outliers automatically.
D. Transform categorical data.

9. A 'Residual Plot' showing a fan shape indicates:

A. Homoscedasticity.
B. Heteroscedasticity.
C. Perfect linearity.
D. Autocorrelation.

10. Logistic Regression is considered a member of:

A. General Linear Models.
B. Generalized Linear Models (GLM).
C. Non-parametric models.
D. Time-series models.

11. The 'Wald Test' in Logistic Regression is used to:

A. Test the significance of individual coefficients.
B. Calculate the R-squared.
C. Verify the normality of residuals.
D. Determine the number of clusters.

12. What is the effect of an outlier on a Linear Regression line?

A. It has no effect if the sample is large.
B. It can significantly pull the line towards itself (influence).
C. It increases the R-squared value always.
D. It changes the model to Logistic.

13. Interaction terms are used in regression to:

A. Simplify the model.
B. Model the case where the effect of one variable depends on another.
C. Replace the intercept.
D. Correct for missing data.

14. Which metric evaluates the trade-off between sensitivity and specificity?

A. F-statistic.
B. AUC-ROC Curve.
C. P-value.
D. Durbin-Watson.

15. In the equation \( y = \beta_0 + \beta_1 x + \epsilon \), \( \epsilon \) represents:

A. The predictor.
B. The random error term.
C. The slope coefficient.
D. The predicted value.

16. Dummy variables are essential when:

A. Including categorical predictors in a model.
B. The dependent variable is continuous.
C. Calculating the mean of a sample.
D. Working with very small datasets.

17. What does a p-value of 0.05 typically signify?

A. There is a 5% chance the null hypothesis is true.
B. The threshold for rejecting the null hypothesis.
C. The model is 95% accurate.
D. The slope is 0.05.

18. Overfitting in regression occurs when:

A. The model is too simple to capture the trend.
B. The model captures noise as if it were a real pattern.
C. The sample size is too large.
D. No independent variables are significant.

19. The Link Function for Logistic Regression is:

A. Identity.
B. Logit.
C. Poisson.
D. Exponential.

20. Stepwise Regression is a method for:

A. Calculating residuals.
B. Automated variable selection.
C. Visualizing data.
D. Normalizing data.

Foundations of Regression Analysis

Professional Overview

Multiple Choice Assessment (20 Questions)

Answer Key