Best Subsets Regression: Model Building and Assumptions
This paper examines the use of the Best Subsets approach within multiple linear regression to identify optimal predictive models for student achievement test scores. Using a dataset of 1,440 students, the analysis evaluates five predictor variables — Curriculum, Household Income, Teacher Experience, Readiness Test, and Pre-Test scores — across stepwise forward regression iterations. The paper applies R-squared and Mean Square Error (MSE) criteria to assess model fit and complexity, demonstrating that different selection criteria can yield different "best" models. Key findings indicate that a four-predictor model and a five-predictor model both merit consideration, depending on the criterion applied. Residual analysis and assumption testing are also discussed.
- Introduction to Predictive Modeling: Defines predictive modeling goals and variable selection
- Best Subsets Observations and Stepwise Results: Stepwise regression results across five predictor models
- Findings: R² and MSE Criteria: Compares R² and MSE to identify best regression model
- Conclusions and Recommendations: Final model recommendation for predicting achievement scores
✍️ How to write this paper — guide, tools & examples ▾
What makes this paper effective
- Clearly defines the evaluation criteria (R² and MSE) before applying them, helping readers understand the rationale behind each model comparison.
- Walks through each stepwise iteration systematically, making it easy to trace how model fit improves as predictors are added.
- Acknowledges the limitation of not using Mallow's Cp-statistic and adjusts the interpretation accordingly, demonstrating intellectual honesty.
Key academic technique demonstrated
The paper demonstrates the use of competing model-selection criteria within the same dataset to show that "best model" is criterion-dependent. By comparing R² and MSE outcomes side by side, the author illustrates that neither criterion alone is definitive, which is a sophisticated point in applied regression analysis.
Structure breakdown
The paper is organized into four sections: a conceptual introduction to predictive modeling, a step-by-step presentation of Best Subsets observations, an analytical findings section comparing R² and MSE results, and a conclusions section summarizing the recommended regression model. The findings section carries the analytical weight, moving from descriptive statistics to interpretive judgment about model selection.
Introduction to Predictive Modeling
The core of predictive modeling is the search for useful predictors. Prediction is centered on a problem defined by the size of the dataset (the number of cases or observations) and the number of potential predictors that can be used to address the problem. A common issue in problem-solving is the enormous number of potential predictors that have only a weak association with the outcome. Computer modeling enables a large number of models to be fit to subsets of the data and tested across additional data subsets. Each test provides an evaluation of the strength of each individual predictor.
The focus of predictive modeling is therefore the search for good subsets of explanatory variables (predictors). Models that fit well with the data are desirable, while models that are a poor fit are not. Moreover, generally speaking, simple models are preferred over complex ones. The process of predictive modeling involves generating a list of useful explanatory variables and, using the data available, fitting many models to the data. The outcome is assessed by evaluating both the simplicity of the models and the fit between the data and the model.
Best Subsets Observations and Stepwise Results
When judging the best subset in a linear regression, the following criteria may be used:
The model with the largest R-squared
The model with the largest adjusted R-squared
The model with the smallest MSE (or S = square root of MSE)
The R-squared criterion and the MSE criterion were used to select the best subset in this analysis. Note that Mallow's Cp-statistic was not used for these observations.
Step 1. Variables entered: 5 Pre-Test (PT). R² = 0.462; MSE = 7,440,136.68.
Step 2. Variables entered: 1 Curriculum (CU) and 5 Pre-Test (PT). R² = 0.803; MSE = 6,469,609.65.
Step 3. Variables entered: 1 Curriculum (CU), 4 Readiness Test (RT), and 5 Pre-Test (PT). R² = 0.863; MSE = 4,637,852.14.
Step 4. Variables entered: 1 Curriculum (CU), 2 Household Income (IN), 4 Readiness Test (RT), and 5 Pre-Test (PT). R² = 0.884; MSE = 3,563,385.51.
Step 5. Variables entered: 1 Curriculum (CU), 2 Household Income (IN), 3 Teacher Experience (TE), 4 Readiness Test (RT), and 5 Pre-Test (PT). R² = 0.884; MSE = 2,850,735.33.
Residuals are shown in the ANOVA as 8,678,396.57. Conducting a Stepwise Forward Regression yielded similar results.
Findings: R² and MSE Criteria
Given that the formula for the R² value is:
R² = SSR / SSTO = 1 − SSE / SSTO
it can be seen that the R² value increases as more variables are added. This means that the "best" model cannot simply be the model with the largest R² value. Instead, R² values are used to identify the point at which adding more predictors no longer makes sense — that is, where additional predictors yield only trivial increases in R². Both the magnitude (number of predictors) and the size of the increase in R² must be considered together.
Using the R² criterion, the increase from the best one-predictor model to the best two-predictor model is substantial: from 0.462 to 0.803. The increase from the two-predictor model to the three-predictor model — from 0.803 to 0.863 — also warrants attention. The increase from the three-predictor model to the four-predictor model is from 0.863 to 0.884, after which the five-predictor model also shows 0.884, indicating no further gain. It would be reasonable to recommend the four-predictor model, although the three-predictor model could also be considered, given that Mallow's Cp-statistic is not being used here to account for bias and variation in predicted responses. Based on this criterion, the best model is the four-predictor model using the predictors CU, IN, RT, and PT.
Mean square error (MSE) measures how far the predicted responses are from the observed responses. Since this difference will be smallest in the best model, the MSE criterion identifies the best regression model as the one with the smallest MSE. Because the MSE value is smallest in the five-predictor model — and the R² value is identical for both the four-predictor and five-predictor models — it would be reasonable to select the five-predictor model as the best, using the predictors CU, IN, TE, RT, and PT. This demonstrates that different criteria can yield different "best" models for prediction.
Always verify citation format against your institution’s current style guide requirements.