Evaluating Model Performance and Insights

1. To begin, collect the subset of data that was NOT used to train the model. You’ll need at least n = 10 unique data points (i.e., values of the independent variables X_i + dependent result variable Y_j). If you’ve got even more than 10 additional points, that’s better.

2. Next, use the model to compute the expected values Y_E,j of the results using the test data points.

3. Calculate the Mean Square Error = 

And then answer the following questions, in the form of an 

1. When you were modeling your data, you attempted to minimize the sum of squared errors (SSE) between the model prediction and the training data. If you divide that SSE by the total number of data points used for fitting, you’ll obtain the MSE for the training data. Compare the MSE for the training data to that of the test data. Which is larger? Speculate as to why this is the case.

2. Think about what the result from part A means for your model’s bias and variance—i.e. what are the strengths and limitations of your model? 

3. What would be the effect of retraining your model, adding the test dataset to the training dataset? Try this, and see what happens. Is the model better off with, or without the additional data?

   Evaluating Model Performance and Insights

   You’ve made it to the end of your research project! It’s now time to look back over your work and attempt to answer your research questions using the analysis that you’ve performed. You may find that your analysis led you to a very clear and concise answer to each research question—if so, that’s great! Utilize your data and visualizations to respond to each question, with at least a paragraph each.

Alternatively, you may have found that your research leaves more questions than it does answers, or that the analysis is not “enough” to answer your questions. You may have even found that your research questions were not relevant, or not answerable at all, given the results! Each of these potentialities is a normal part of the research process, and not something that should worry you. Rather, it is simply an indication that more work needs to be done! If you find that you’re in this latter category, you should see if you can come up with alternative research questions that you can answer, or otherwise re-interpret your research findings through the lens of your data and analysis. It would also be a good idea to remark on how your perspective of the problem has shifted over the course of the project. 

Evaluate your questions and findings in the  Be sure to make direct reference to your data and visualizations, and provide solid evidence to support your conclusions. APA