Traditional wisdom in machine learning holds that there is a careful trade-off between training error and generalization gap. There is a "sweet spot" for the model complexity such that the model (i) is big enough to achieve reasonably good training error, and (ii) is small enough so that the generalization gap - the difference between test error and training error - can be controlled. A smaller model would give a larger training error while making the model bigger would result…
