Using held-out dev data is the best way to do model selection. See http://www.econ.upf.edu/~lugosi/esaimsurvey.pdf: “But whatever promising the pre-testing method may look like, it will be hard to convince practitioners to abandon cross-validation and other resampling methods. Indeed, a straightforward analysis of the hold-out approach to model selection suggests that hold-out enjoys almost all the desirable features of any foreseeable model selection method.”