Topological and Dynamical Complexity of Random Neural Networks

Gilles Wainrib1, and Jonathan Touboul2, 3,

arXiv:1210.5082v2 [math-ph] 6 Nov 2012

Random neural networks are dynamical descriptions of randomly interconnected neural units. These show a phase transition to chaos as a disorder parameter is increased. The microscopic mechanisms underlying this phase transition are unknown, and similarly to spin-glasses, shall be fundamentally related to the behavior of the system. In this Letter we investigate the explosion of complexity arising near that phase transition. We show that the mean number of equilibria undergoes a sharp transition from one equilibrium to a very large number scaling exponentially with the dimension on the system. Near criticality, we compute the exponential rate of divergence, called topological complexity. Strikingly, we show that it behaves exactly as the maximal Lyapunov exponent, a classical measure of dynamical complexity. This relationship unravels a microscopic mechanism leading to chaos which we further demonstrate on a simpler class of disordered systems, suggesting a deep and underexplored link between topological and dynamical complexity.
PACS numbers: 87.18.Tt, 05.10.-a 87.19.ll, 87.18.Sn, 87.18.Nq

Heterogeneity of interconnections is a crucial property to understand the behavior of realistic networks that arise in the modeling of physical, biological or social complex systems. Among these, a paramount example is given by neuronal networks of the brain. In these systems, synaptic connections display characteristic disorder properties [1, 2] resulting from development and learning. Taking into account this heterogeneity seems now essential, as experimental studies of neuronal tissues have shown that the degree of disorder in the connections signicantly impacts the input-output function, rhythmicity and synchrony, eects that can be related to transitions between physiological and pathological behaviors. As an example, Aradi and Soltesz [3] have shown that rats subject to febrile seizure present the same average synaptic properties but increased variance. These properties are reminiscent of disordered physical systems such as spin-glasses. The relationship between disorder and qualitative behaviors has been thoroughly studied within theoretical frameworks such as the Sherrington-Kirkpatrick model [4] describing the behavior of binary variables interacting through a random connectivity matrix. In these models, estimating the number of metastable states, deemed to be deeply related to the transition to chaos, remains an important endeavor [5]. In the context of nervous system modeling, random neural networks [6, 7] constitute a prominent class of models, which describe the evolution of the activity of a neuron i in a n-neurons network through the randomly coupled system of ordinary dierential equations:
n

x i = xi +
j =1

Jij S (xj )

(1)

where Jij are independent centered Gaussian random variables of variance 2 /n representing the synaptic connectivity coecients between neuron i and j , and S is

an odd sigmoid function with maximal unit slope at the origin (S (x) S (0) = 1) representing the synaptic nonlinearity. The behavior of system (1) has been analyzed in the asymptotic regime of innite population size n [6] and displays a phase transition for a critical value of the disorder at = 1: for < 1 all the trajectories are attracted to the trivial equilibrium x = 0 and for > 1 the trajectories have chaotic dynamics. This generic phase transition has been numerically observed in a number of situations in more realistic models involving multiple populations and excitable dynamics [8]. That phase transition, and the chaotic regime beyond the edge of chaos, appear particularly relevant to understand the computational capabilities of neuronal networks. In particular, information processing capacity was characterized as optimal at the edge of criticality [9, 10], and such random neural networks are in particular used in recent machine learning algorithms [11]. Moreover, the question of criticality has been widely debated in the theoretical neuroscience community and beyond [1214] and it seems that through a number of dierent mechanisms among which plasticity of synapses, networks may tend to be naturally poised near criticality. In the random neural network, the microscopic mechanisms underpinning this phase transition remains unclear. It is hence of great interest to dissect precisely the behavior of such systems at the edge of chaos. This question of the topological modications of the phase space arising at the edge of chaos is precisely the question we shall address in this Letter. More precisely, we will characterize the averaged number of equilibria in the random neural network (1). To this end, we will develop upon the theories of random matrices [15], random geometry and Gaussian elds [16]. We denote by An ( ) the random number of equilibria

2 (depending on the realization of the matrix J = (Jij )). These are the solutions of the system:
n

xi =
j =1

Jij S (xj )

(2)

For < 1, consistently with the mean-eld analysis, we rst show that E [An ( )] 1 when n . The proof proceeds by showing that the system is contracting on the whole space Rn . To this purpose, one needs to characterize the eigenvalues of the random Jacobian matrix I + J.(S (x)) where I is the identity matrix, and (S (x)) is the diagonal matrix with elements S (xi ). The matrix J.(S (x)) is a centered Gaussian matrix with independent components, and each column has a distinct standard deviation given by 2 S (xi )2 /n. Rajan and Abbott [17] provide the system of equations satised by the squared modulus of the eigenvalues of such random matrices, and solve these when considering only two dierent variances. Their methodology readily generalizes to our problem, and elementary algebraic manipulations (see [18, 19]) show that the spectral density has 2 n 2 support in the disc of radius i=1 S (xi ) for large n n. In our case, S (x) 1, and hence all eigenvalues of the matrix I + J (S (x)) have a negative real part in the limit n . This implies global contraction of the dynamics, ensuring the fact that the trivial equilibrium x = 0 is the unique equilibrium for < 1 (Banach xed point theorem) and its global stability. For > 1 the situation is more complex. Similarly to the phase transition in spin glasses, the behavior of E [An ( )] is likely to scale as exp(n C ( )) where C ( ) is the topological complexity. Our starting point is to observe that the xed point equations (2) can be viewed as zero crossings of a Gaussian eld indexed by x Rn , and therefore we may apply the Kac-Rice [16] formula in order to estimate the number of solutions: E [An ( )] =
Rn

equilibria taking advantage of the fact that all equilibria remain close to x = 0. More precisely, we rst show that with an arbitrarily high probability 1 (n, ) with (n, ) 0 as n , all equilibria belong to a ball B() centered at x = 0 of radius () which tends to 0 as 0. This is a consequence of the spectral analysis of the random matrix I + J (S (x)), whose eigenvalues all have negative real parts outside of B() for large n. The property that xed points remain in a small ball around zero is non-trivial. The proof proceeds by dening () the unique positive solution of the scalar equation x/ = S (x), which is clearly arbitrarily small when 0, and the smooth modied sigmoid function S, S, (x) = x/ |x| < () S (x) |x| > () +

(the small interval [(), () + ] allows to dene a smooth continuation). Because of the properties of the sigmoidal function, and in particular the fact that the dierential is decreasing for x > 0 (and increasing for x < 0), the same argument as used in the case < 1 applied to system (1) dened with the sigmoid S, (termed modied system ) ensures that all eigenvalues of I + J (S, (x)) have negative real part and hence that the unique xed point of the modied system is 0. In particular, there is no xed point outside the ball of radius (). In that region the original and modied system are identical, implying that the only possible xed points of the original system are contained in the ball of radius (). Therefore, one can split the expectation according to whether |x| < () or not, yielding: E [An ( )] =
B()

dx

| det(I + J.(S (x)))|

0 (x + J.S (x)) + O( (n, )) dx | det(I + J.(S (x)))| 0 (x + J.S (x)) . Recent studies [2022] have used the Kac-Rice formula to estimate the mean number of critical points in random energy landscapes arising from Hamiltonian spinglass with strong symmetry properties. In our case, the situation is substantially dierent: (i) there is no underlying energy landscape since the system (1) is not Hamiltonian and (ii) symmetry properties are relatively weak and do not enable the same kind of reduction method developed in [2022]. A major technical diculty is the fact that, because the lack of symmetry in our system, one needs to deal with determinant of random matrices with columns of non-identical variances, for which the spectral density is unknown. However, near criticality, for = 1 + with 0 < 1, we can obtain a rst order estimate of the number of Moreover, thanks to the dierentiability of the determinant operator, we know that within the ball B() , the integrand is equal to | det(I + J)| + O(()), eventually yielding: E [An ( )] = E [| det(I + J)|] + O(() + (n, )) To evaluate this formula, we rst compute the logarithm of the determinant: 1 1 log | det(I + J)| = n n log | 1|
sp(J)

where sp(J) denotes the spectrum of J, which in the large n limit is uniformly distributed in the disc of radius [23]. Using this property one obtains in the large n limit 1 log | det(I + J)| = c( ) + R(n) n

3 and c( ) := C log |z 1| (dz ) with (dz ) the uniform distribution on the disk of radius = 1 + and R(n) is the nite-size error associated to the convergence to the circular law. We have: 1 1 log E [| det(I + J)|] = c( ) + log E enR(n) . (3) n n
1 log E [An ( )] is arbitrarily close c( ) This ensures that n when and n . We are hence left computing c( ). Since log |z | is harmonic, we can show that for a, b > 0 : 2

equation. Near criticality, the decomposition dramatically simplies and yields for 1+ : ( 1)2 . We hence conclude that the topological and dynamical complexities have the same behavior at the edge of chaos. More surprising is that using the spectral decomposition of the Schr odinger equation in the limit 1, Sompolinsky and collaborators show that diverges as log( ), precisely as the complexity of the system given in formula (4), although our analysis rigorously only applies for close to 1. This further strengthens our microscopic interpretation of the emergence of chaos in relationship with the number of saddles. If this interpretation is the actual phenomenon arising in random neural networks, then simpler, lowerdimensional dynamical systems shall display a similar relationship between the number of unstable equilibria and the Lyapunov exponent. Probably the simplest model one could think of is what we call the fakir bed dynamics, corresponding to the movement of a particle in a two-dimensional complex landscape with k unstable xed points. More precisely, we now consider a particle conned in a compact subset of R2 , with close to the origin a xed number k of Gaussian hills (corresponding to the presence of k unstable xed points) randomly chosen in space. Trajectories of particles soon get chaotic as k increases (see Fig. 1(b)), and we numerically compute the maximal Lyapunov exponent of the trajectories. Since the landscape is probabilistic due to the choice of the location of hills, we compute the averaged maximal Lyapunov exponent across 100 independent realizations of the process, and plot it against the logarithm of the number of equilibria. The corresponding curve (Fig. 1(a)), indeed displays an increasing prole well approximated by a linear curve: a clear relationship again emerges between number of unstable xed points and Lyapunov exponents, supporting our interpretation related to the random neural network.
0.5 0.45

log |a bei |d = 2 log(min(a, b))

ensuring that c( ) = 0 for < 1, which is consistent with our previous analysis of this case, and c( ) = 1 2

2r log(r) dr
1

= log( ) +

1 2

1 1 2

(4)

for > 1. For close to 1+ , we conclude c( ) ( 1)2 . Therefore we have shown that: [An ( )] en(1)
2

up to multiplicative polynomial factors. This combinatorial explosion of the number of equilibria is the hallmark of the accumulation of bifurcations [29] in a the neighborhood of the critical parameter value = 1. Moreover, this result provides a possible topological explanation for the emergence of chaos. For > 1, the phase space is heavily mined with equilibrium points, most of which are saddles due to the spectral properties of the Jacobian matrices as discussed above. Typical trajectories evolving in this landscape will hence wander from the vicinity of one point to another following stable manifolds. The presence of a large number of saddle points induces a very high sensitivity to perturbations, distinctive feature of chaos. The classical characterization of chaos relies on the evaluation of the maximal Lyapunov exponent of the trajectories quantifying the dynamical complexity. This quantity is dened as follows. Applying an innitesimal external perturbation xi xi + x0 i on neuron i at time t0 induces a change on all neurons at subsequent times xj (t) xj (t) + xji (t) dening a susceptibility matrix ij (t0 + , t0 ) = xji (t0 + )/x0 i . From this matrix we shall dene an averaged susceptibility 1 2 2 ( ) = limt0 n E i,j ij (t0 + , t0 ) and eventually dene the Lyapunov exponent by: = lim log(2 (t)) . t t

0.4

0.35

0.3

0.25

0.2

0.15

0.1 0 10

10

10

(a)Lyapunov exponent vs number of xed point

(b)Fakir bed trakectories

FIG. 1. The Fakir bed dynamics. (a): blue: Lyapunov exponent averaged over 100 realizations of the Fakirs dynamics vs number of xed points (in semilogarithmic axes) and linear regression. (b): Red and Green: two trajectories with very close initial conditions.

This quantity was analyzed in [6] using a spectral decomposition based on the one-dimensional Schr odinger

In this letter, motivated by the analysis of ne microscopic phenomena arising at the edge of chaos in random

4 neural networks, we have shown that the average number of equilibria scales exponentially with the system size, with an exponential coecient proportional to the Lyapunov exponent. This property, relying in part on the spectral theory of random matrices, readily inherits universality properties of the circular law [25], and therefore is valid for a large class of independent couplings beyond Gaussian. The complexity was shown to vary smoothly with the disorder parameter with a critical exponent 2, larger than 1, ensuring a form of structural robustness in the neighborhood of the phase transition. This study is the rst application to random neural networks of recent methods used for counting the number of metastable equilibria in spin glasses. From the theoretical viewpoint, we extended that approach to out of equilibrium, non-Hamiltonian systems at zero temperature (singular points of a vector eld). Incidentally, the relationship found between topological and dynamical complexity highlighted what we conjecture to be a deep correspondence. Numerous open questions and perspectives have emerged from this study, among which the estimation of the distribution of equilibria and their number beyond the edge of criticality, requiring signicant advances in the analysis of random matrix determinants, or pursuing the exploration, in line with [2628], of the relationship between topological and dynamical complexity with other measures such as the fractal dimensions of chaotic attractors.
ters 35, 1792 (1975). [5] G. Parisi, Les Houches 83, 295 (2006). [6] H. Sompolinsky, A. Crisanti, and H. Sommers, Physical Review Letters 61, 259 (1988). [7] S. Amari, Syst. Man Cybernet. SMC-2 (1972). [8] G. Hermann and J. Touboul, Physical Review Letters 109, 018702 (2012). [9] T. Toyoizumi and L. Abbott, Physical Review E 84, 051908 (2011). [10] D. Sussillo and L. Abbott, Neuron 63, 544 (2009). [11] H. Jaeger and H. Haas, Science 304, 78 (2004). [12] P. Bak, How nature works: the science of self-organized criticality, Vol. 212 (Copernicus New York, 1996). [13] J. Beggs, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 366, 329 (2008). [14] M. Kitzbichler, M. Smith, S. Christensen, and E. Bullmore, PLoS Computational Biology 5, e1000314 (2009). [15] T. Tao, Topics in random matrix theory, Vol. 132 (Amer Mathematical Society, 2012). [16] R. Adler and J. Taylor, Random elds and geometry, Vol. 115 (Springer Verlag, 2007). [17] K. Rajan and L. Abbott, Physical Review Letters 97, 188104 (2006). [18] L. Garcia del Molino, Dynamics of randomly connected systems with applications to neural networks, Masters thesis, Ecole Polytechnique (2012). [19] Y. Wei, Physical Review E 85, 066116 (2012). [20] Y. Fyodorov, Physical Review Letters 92, 240601 (2004). [21] Y. Fyodorov and I. Williams, Journal of Statistical Physics 129, 1081 (2007). [22] A. Aunger and G. Arous, Arxiv preprint arXiv:1110.5872 (2011). [23] V. Girko, Theory of Probability and its Applications 29, 694 (1985). [24] B. Cessac, B. Doyon, M. Quoy, and M. Samuelides, Physica D: Nonlinear Phenomena 74, 24 (1994). [25] T. Tao and V. VAN, Communications in Contemporary Mathematics 10, 261 (2008). [26] F. Ledrappier, The Annals of Mathematics 122, 540 (1985). [27] P. Frederickson, J. Kaplan, E. Yorke, and J. Yorke, Journal of Dierential Equations 49, 185 (1983). [28] S. Chatterjee, arXiv:0810.4221. [29] B. Cessac, Journal de Physique I 5, 409 (1995).

[1] [2] [3] [4]

[email protected] [email protected] D. Parker, The Journal of Neuroscience 23, 3154 (2003). E. Marder and J. Goaillard, Nature Reviews Neuroscience 7, 563 (2006). I. Aradi and I. Soltesz, The Journal of Physiology 538, 227 (2002). D. Sherrington and S. Kirkpatrick, Physical Review Let-