Margin Maximization for Robust Classification using Deep Learning

Alexander Matyasko, Lap‑Pui Chau · IJCNN 2017

Abstract

Deep neural networks have achieved significant success for image recognition problems. Despite the wide success, recent experiments demonstrated that neural networks are sensitive to small input perturbations, or adversarial noise. The lack of robustness is intuitively undesirable and limits neural networks applications in adversarial settings, and for image search and retrieval problems. Current approaches consider augmenting training dataset using adversarial examples to improve robustness. However, when using data augmentation, the model fails to anticipate changes in an adversary. In this paper, we consider maximizing the geometric margin of the classifier. Intuitively, a large margin relates to classifier robustness. We introduce novel margin maximization objective for deep neural networks. We theoretically show that the proposed objective is equivalent to the robust optimization problem for a neural network. Our work seamlessly generalizes SVM margin objective to deep neural networks. In the experiments, we extensively verify the effectiveness of the proposed margin maximization objective to improve neural network robustness and to reduce overfitting on MNIST and CIFAR-10 dataset.

TL;DR

Generalizes SVM margin objective to deep networks, which theoretically equivalent to robust optimization and empirically improves robustness on MNIST/CIFAR‑10.

SVM geometric margin illustration — SVM maximizes the geometric margin in input space, improving robustness to small perturbations.

Deep network margin concept — Our method maximizes the geometric margin of deep networks in input space by reducing the cross-class input sensitivity.

Core Idea

Intuitively, a large geometric margin implies greater classifier robustness. To enhance the robustness of deep neural networks (DNNs), a natural approach is to maximize the geometric margin in input space. For linear classifiers, the standard SVM objective explicitly maximizes this margin and is equivalent to a robust optimization problem. Applied directly to deep networks, however, the SVM objective maximizes the margin in feature space. Because DNN feature mappings are highly non‑linear, a large feature‑space margin does not necessarily imply a large input‑space margin and may fail to improve robustness.

To estimate the geometric margin for a non‑linear classifier such as a DNN, we linearize the model locally. Let $f:\mathbb{R}^d\to\mathbb{R}$ be a non‑linear binary classifier with the decision boundary $f(\mathbf{x})=0$. We estimate the distance to the boundary using a first‑order Taylor approximation of the function level set:

$$ f(\mathbf{x}) + \nabla_{\mathbf{x}} f(\mathbf{x})^{\top} \mathbf{r} = 0 $$

Using this approximation, the geometric margin $\hat{\gamma}$ can be calculated in the closed‑form:

$$ \hat{\gamma} = \frac{|f(\mathbf{x})|}{\|\nabla_{\mathbf{x}} f(\mathbf{x})\|_2} $$

For multiclass DNNs, we view the model as a collection of pairwise binary classifiers. The boundary between classes $y$ and $j$ is given by $f_y(\mathbf{x}) = f_j(\mathbf{x})$. The corresponding geometric margin between classes $y$ and $j$ can be approximated as:

\[ \hat{\gamma}_{y,j}(\mathbf{x}) = \frac{|f_y(\mathbf{x})-f_j(\mathbf{x})|} {\|\nabla_{\mathbf{x}} f_y(\mathbf{x})-\nabla_{\mathbf{x}} f_j(\mathbf{x})\|_q}, \quad j\neq y,\qquad \hat{\gamma}(\mathbf{x})=\min_{j\neq y}\gamma_{y,j}(\mathbf{x}) . \]

We refer to the numerator as the functional margin and to the denominator as the margin sensitivity. Margin sensitivity measures how quickly the functional margin changes as we move toward the decision boundary. Penalizing this sensitivity directly increases the input‑space margin.

Analogous to the soft‑margin SVM, we propose soft‑margin maximization objectives for deep networks:

Binary Margin Maximization: \[ \min_{\theta}\;\frac{1}{m}\sum_{i=1}^m \big(1-y_i f(\mathbf{x}_i)\big)_+ \;+\; C\,\|\nabla_{\mathbf{x}} f(\mathbf{x}_i)\|_q . \tag{B} \]
Multiclass Margin Maximization: \[ \min_{\theta}\;\frac{1}{m}\sum_{i=1}^m \max_{j\neq y_i}\big(1+f_j(\mathbf{x}_i)-f_{y_i}(\mathbf{x}_i)\big)_+ \;+\; C\,\epsilon(\mathbf{x}_i,y_i), \tag{M} \] \[ \epsilon(\mathbf{x},y)=\max_{j\neq y}\big\|\nabla_{\mathbf{x}} f_y(\mathbf{x}) -\nabla_{\mathbf{x}} f_j(\mathbf{x})\big\|_q . \]

For deep linear networks, our formulation reduces to the standard soft‑margin SVM. Similar to SVM, we show that the proposed objectives are equivalent to robust optimization problems with $\ell_p$‑norm uncertainty sets (Theorems IV.1 and IV.2). Compared to adversarial training (AT), our margin‑maximization method is attack‑agnostic yet attacker‑aware: it penalizes the directions attackers can exploit (steep gradients of logit differences), improving clean generalization and average robustness.

On MNIST and CIFAR‑10, our margin objective reduces overfitting, increases the average margin $\bar{\gamma}$, and improves robustness against the DeepFool attack. At the same perturbation budget, adversarial examples become semantically more distinct, which is consistent with larger input‑space margins.

Adversarial examples generated using DeepFool for Lenet-5 on MNIST — Adversarial examples generated using DeepFool for convolutional neural network Lenet-5 on MNIST dataset. Rows show original images (top), adversarial images (middle), and adversarial images after applying our regularization (bottom). As a result of our regularization, the adversarial images become visually distinguishable from the originals.

How to Cite

@inproceedings{matyasko2017margin,
  title         = {Margin maximization for robust classification using deep learning},
  author        = {Matyasko, Alexander and Chau, Lap{-}Pui},
  year          = {2017},
  month         = {May},
  booktitle     = {2017 International Joint Conference on Neural Networks (IJCNN)},
  pages         = {300--307},
  doi           = {10.1109/IJCNN.2017.7965869}
}