The idea for training neural networks with SVM loss came from Ronan Collobert's 2004 PhD Thesis and this paper: Collobert 2004 which has some conceptual errors (mainly that SVM loss is not Perceptron loss).