publications

2023

1. 

   On the Varied Faces of Overparameterization in Supervised and Self-Supervised Learning

   Matteo Gamba, Arna Ghosh, Kumar Krishna Agrawal, and 3 more authors

   NeurIPS workshop SSL Theory and Practice, 2023

   Abs PDF Website

   The quality of the representations learned by neural networks depends on several factors, including the loss function, learning algorithm, and model architecture. In this work, we use information geometric measures to assess the representation quality in a principled manner. We demonstrate that the sensitivity of learned representations to input perturbations, measured by the spectral norm of the feature Jacobian, provides valuable information about downstream generalization. On the other hand, measuring the coefficient of spectral decay observed in the eigenspectrum of feature covariance provides insights into the global representation geometry. First, we empirically establish an equivalence between these notions of representation quality and show that they are inversely correlated. Second, our analysis reveals the varying roles that overparameterization plays in improving generalization. Unlike supervised learning, we observe that increasing model width leads to higher discriminability and less smoothness in the self-supervised regime. Furthermore, we report that there is no observable double descent phenomenon in SSL with non-contrastive objectives for commonly used parameterization regimes, which opens up new opportunities for tight asymptotic analysis. Taken together, our results provide a loss-aware characterization of the different role of overparameterization in supervised and self-supervised learning.

2. 

   On the Lipschitz Constant of Deep Networks and Double Descent

   Matteo Gamba, Hossein Azizpour, and Mårten Björkman

   In Procedings of the British Machine Vision Conference 2023, 2023

   Abs PDF Code Poster Slides

   Existing bounds on the generalization error of deep networks assume some form of smooth or bounded dependence on the input variable, falling short of investigating the mechanisms controlling such factors in practice. In this work, we present an extensive experimental study of the empirical Lipschitz constant of deep networks undergoing double descent, and highlight non-monotonic trends strongly correlating with the test error. Building a connection between parameter-space and input-space gradients for SGD around a critical point, we isolate two important factors – namely loss landscape curvature and distance of parameters from initialization – respectively controlling optimization dynamics around a critical point and bounding model function complexity, even beyond the training data. Our study presents novels insights on implicit regularization via overparameterization, and effective model complexity for networks trained in practice.

3. 

   Deep Double Descent via Smooth Interpolation

   Matteo Gamba, Erik Englesson, Mårten Björkman, and 1 more author

   In Transactions in Machine Learning Research, 2023

   Abs PDF Code

   The ability of overparameterized deep networks to interpolate noisy data, while at the same time showing good generalization performance, has been recently characterized in terms of the double descent curve for the test error. Common intuition from polynomial regression suggests that overparameterized networks are able to sharply interpolate noisy data, without considerably deviating from the ground-truth signal, thus preserving generalization ability. At present, a precise characterization of the relationship between interpolation and generalization for deep networks is missing. In this work, we quantify sharpness of fit of the training data interpolated by neural network functions, by studying the loss landscape w.r.t. to the input variable locally to each training point, over volumes around cleanly- and noisily-labelled training samples, as we systematically increase the number of model parameters and training epochs. Our findings show that loss sharpness in the input space follows both model- and epoch-wise double descent, with worse peaks observed around noisy labels. While small interpolating models sharply fit both clean and noisy data, large interpolating models express a smooth loss landscape, where noisy targets are predicted over large volumes around training data points, in contrast to existing intuition.

2022

1. 

   Overparameterization Implictly Regularizes Input Smoothness

   Matteo Gamba, Hossein Azizpour, and Mårten Björkman

   NeurIPS workshop Interpolate, 2022

   PDF Poster

2. 

   Are All Linear Regions Created Equal?

   Matteo Gamba, Adrian Chmielewski-Anders, Josephine Sullivan, and 2 more authors

   In International Conference on Artificial Intelligence and Statistics, 2022

   Abs PDF Code Poster Slides

   The number of linear regions has been studied as a proxy of complexity for ReLU networks. However, the empirical success of network compression techniques like pruning and knowledge distillation, suggest that in the overparameterized setting, linear regions density might fail to capture the effective nonlinearity. In this work, we propose an efficient algorithm for discovering linear regions and use it to investigate the effectiveness of density in capturing the nonlinearity of trained VGGs and ResNets on CIFAR-10 and CIFAR-100. We contrast the results with a more principled nonlinearity measure based on function variation, highlighting the shortcomings of linear regions density. Furthermore, interestingly, our measure of nonlinearity clearly correlates with model-wise deep double descent, connecting reduced test error with reduced nonlinearity, and increased local similarity of linear regions.

2020

1. 

   Hyperplane arrangements of trained convnets are biased

   Matteo Gamba, Stefan Carlsson, Hossein Azizpour, and 1 more author

   arXiv preprint arXiv:2003.07797, 2020

   Abs PDF Code

   We investigate the geometric properties of the functions learned by trained ConvNets in the preactivation space of their convolutional layers, by performing an empirical study of hyperplane arrangements induced by a convolutional layer. We introduce statistics over the weights of a trained network to study local arrangements and relate them to the training dynamics. We observe that trained ConvNets show a significant statistical bias towards regular hyperplane configurations. Furthermore, we find that layers showing biased configurations are critical to validation performance for the architectures considered, trained on CIFAR10, CIFAR100 and ImageNet.

2019

1. 

   On the Geometry of Rectifier Convolutional Neural Networks

   Matteo Gamba, Hossein Azizpour, Stefan Carlsson, and 1 more author

   In Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019

   Abs PDF Code Poster

   While recent studies have shed light on the expressivity, complexity and compositionality of convolutional networks, the real inductive bias of the family of functions reachable by gradient descent on natural data is still unknown. By exploiting symmetries in the preactivation space of convolutional layers, we present preliminary empirical evidence of regularities in the preimage of trained rectifier networks, in terms of arrangements of polytopes, and relate it to the nonlinear transformations applied by the network to its input.