Exploring the Neural Network-Quantum Field Theory Correspondence

Abstract

Signal propagation is an important factor impacting the trainability of neural networks where better signal propagation leads to better training performance. Signal propagation can be ordered, where signals decay, chaotic, where signals decorrelate or critical, where a signal neither decays nor decorrelates. Mean Field Theory (MFT) based formulations exist to predict signal propagation in neural networks, which have shown empirical success in achieving better training performance. MFT formulations, however, operate under the assumption of infinitely wide network layers, which do not exist in practice. Quantum field theory-based formulations aim to correct this assumption by formulating a new signal propagation prediction called the NN-QFT correspondence. In this thesis, the accuracy of predictions from the NN-QFT correspondence have been empirically explored. The NN-QFT correspondence appears to be data-invariant for feedforward neural networks and accurate in predictions for networks sufficiently distanced from the critical point. For linear networks, the theory appears to predict the critical point correctly. The theoretical critical point for nonlinear networks does not match the empirically found critical point. Critical behaviour observed for these networks appears to agree with theory. Additionally, observations appear to correlate the periodic behaviour of individual neurons and the critical to chaotic transition.