BULLETIN OF MATHEMATICAL BIOLOGY

VOLUME 38, 1976

CONNECTIVITY IN RANDOM NETWORKS

[] DEREK F. STUBBSand Pmi,Ln, I. Good The Upjohn Company, Kalamazoo, Michigan 49001, U.S.A.

Applications of r a n d o m l y c o n n e c t e d n e t w o r k s are r e v i e w e d briefly. T h e c o n n e c t i v i t y of a r a n d o m n e t w o r k h a s b e e n defined i n a v a r i e t y of w a y s i n c l u d i n g o u t p u t c o n n e c t i v i t y , total or n e t w o r k c o n n e c t i v i t y , c o n n e c t a n c e , e x p e c t e d p a t h l e n g t h a n d radius. O n e or more of t h e s e definitions m a y p r o v e m o r e c o n v e n i e n t in a g i v e n e x p e r i m e n t a l s y s t e m . I n t e r r e l a t i o n s a m o n g t h e s e definitions are d e r i v e d a n d d i s p l a y e d a n d a s y m p t o t i c r e s u l t s p r o v i d e d i n t h e f o r m of t w o t h e o r e m s . C o m p u t e r s i m u l a t i o n s were u s e d to explore t h e range of a p p l i c a t i o n of t h e s e a s y m p t o t i c a p p r o x i m a t i o n s . T h e r e s u l t s were u s e d to d e t e r m i n e the o u t p u t c o n n e c t i v i t y of t h e n e u r o n s of t h e b r a i n .

Introduction.

The concept of a randomly connected network through which mass, energy or information flows has been widely applied. Examples include: the spread of excitation in cardiac muscle (Wiener and Rosenbluth, 1946), neural networks (see the reviews b y Rashevsky, 1960, and Stubbs, 1975), the spread of a rumor or a contagious disease in a population (Rapoport, 1953a, b, 1954; Cane, 1966; Daley, 1967), and the spread of cancer in an organism (Blumenson, 1970). Randomly connected networks have been applied to problems in such diverse areas as structural chemistry (Correia and Goveia, 1971; Bell and Dean, 1972; Alben and Boutron, 1975), sociology (Doreian, 1974) and information retrieval (Schultz and Hubert, 1973). The response of a network to a stimulus depends upon its structure. We consider one important aspect of network structure here--the pattern of connections between nodes. The connectivity of random networks has been defined in different ways. Shimbel (1948) and Rapoport (1948) introduced the concept of output connectivity, the number of connections sent out b y each individual node in a network. In particular, they studied the number of axons sent out by individual neurons in a neural network. Total or network connectivity 295

296

DEREKF. STUBBS AND PHILLIP I. GOOD

was defined by Solomonoff and Rapoport (1951) as the fraction of nodes in a network which could be reached in one or more steps from a given node. Gardner and Ashby (1970) introduced the concept of connectance, the proportion of nodes to which any one node is directly connected. They showed that the stability of a network is directly related to its connectance. Their work developed from an earlier paper on the stability of random neural networks by Ashby, yon Foerster and Walker (1962) from which a series of communications stemmed dealing with ecosystems (May, 1972; Siljak, 1974; Roberts, 1974), metabolic networks (Kauffman, 1969 ; Newman, 1972 ; Park, 1975), neural nets (Grffiith, 1963a) and some general theoretical observations (Fitzhugh, 1963; Walker and Ashby, 1966; Somorjai and Goswami, 1972; McMurtrie, 1975). The expectedpath length or mean number of steps between any two nodes of a network is also a measure of connectivity as is the radius, a concept familiar to students of graph theory (Wilson, 1972, p. 31). Our objective is to determine the interrelations among the various definitions of connectivity. I n the next section we present formal definitions and two theorems dealing with asymptotic relationships. In Section 3 we evaluate the accuracy of these relationships when the number of nodes is finite. Finally, in Section 4 we present an application of our theorems, using them to circumvent experimental obstacles encountered in the study of neural networks in the brain.

Basic Definitions. We define a network as the couple (N, M) consisting of a set of N nodes and an associated N × N connectivity matrix M. There are two basic classes of networks. The first entails a deterministic selection mechanism of the selection of M and is examined in classical graph theory. The second, using combinatorial theory as in Shimbel (1948) and Rapoport (1948), entails a probabilistic selection mechanism for the selection of M. We study this second class. We define the output connectivity of t h e / t h node of a network (N, M) as the sum of the elements of the I t h row of the matrix M and the outdegree (or outvalency) as the number of nonzero elements in the I t h row. We confine our attention in what follows to random networks of the form (N, #[a], P) where N is a countable set of nodes; #[a] is a set of/V × N matrices whose row sums are equal to a constant a; P is a probability measure defined on the smallest sigma field containing all subsets of #[a]; #[a] is closed and P is invariant under all permutations of rows and columns. Note t h a t the outdegree may vary from node to node while the output connectivity is fixed at a. Griffith (1963b, 1965) considered noncountable sets of nodes. Nonsymmetric random networks were studied by Rapoport (1948), Shimbel (1948) and Harmon (1964). Variable output connectivity was considered by Farley (1964), Variable output connectivity was considered by Farley (1964) and by Anninos and Elul (1974).

CONNECTIVITY

I_ATR A N D O M

NETWORKS

297

We will say t h a t node I is connected to node J in m steps, d(I, J) = in, where I and J are distinct nodes in (N, M), if there exists an integer n, such t h a t the I J t h element of the matrix M n is nonzero and in is the smallest such integer. We will say t h a t two nodes are connected if they are connected in some number of steps and set d(I, J) = co otherwise. We define the connectivity of a network (N, M) for a node I as t h e proportion of nodes to which node I is connected. We define 7, the connectivity of a random network, as the expected value . (with respect to P) of this proportion. We define the path length of a network (N, M) for a node I as the average number of steps to each node with which it is connected. We define L, the expected path length between two connected nodes of a random network, as the expected value of this average. Because # is closed and P is invariant under permutations of the rows and columns, neither 7 nor L depend upon node I. We define r the radius of a network (N, M) as mini maxj d(i, j) and the radius of ~ random network as minu min~ maxj d(i, j). Discussions of such random networks have been divided into the case of unit output connectivity (Shimbel, 1948; Rapoport, 1948) and the case of m.ultiple output connectivity (Solomonoff and Rapoport, 1951). We will consider each of these eases in turn.

The Case of Unit Output Connectivity. Consider a given neuron, say A. In exactly 1/N of all networks it is connected only to itself. In M ( N - 1)!~(NM) !Nm of all networks it is connected to no more than M nodes, N 1

=

N--1

Z ( M ( N - 1 ) ! ) I ( ( N - M ) ! N M) = ( N - 1)IN -N ~ N M ( N I M ) / M ! , M=I

M=O,

(1) for N-I

Z NM(N-M)/M!

N--I

= N Z

M=O

N--2

NM/M!-N Z

M=O

= NN/(N.

NM/M!

M=O

1)!

(2)

By definition, N-1

7 = ( ( N - 1)!N--N))/N ~ I y M ( N - M ) ( N - , M ) / M ! ,

(3)

M=0

and N-1

L = ( N - 1)!N -N ~, N M ( N - M ) ( ( N - M - 1)/2)/M!,

(4)

M=O

for if A is linked out to exactly M nodes, then the average path length is M ( M - 1 + 0)/2M.

298

DEREK

F. STIJBBS A N D P H I L L I P

I. G O O D

THEORE~ 1 : I n a random network of N nodes with unit output connectivity, as N --+ ~ , ? ~ %/7c/2N, and L ~ %/~zN/8. Proof: /V--1

N-1

NM(M + M(M - 1)-NM+N(N-M))/M!

NM(N- M)2/M! = M=0

M=O

N-1

? = N!N-!V-1 ~ NM/M!-(N-1)/N+I. M=0

Now 1 I ~ exp ( - t2/2) dt = ½ lim e -z¢ iv=2 ~ NM/M! N~ M=0 ~/2N o Using Stirling's formula, N IN -N = ~/2uN. exp ( - N), ? ---* %/~/2N, as asserted.

(5)

(Note that Shimbel (1948) and Rapoport (1948) obtained this result in a different context). From equation (4) N--1

L = ( N - 1 ) I N -N ~ N M [ ( N - M ) 2 / 2 - ( N - M ) / 2 ] / M ! M=0 .N-2

= N!N-~V ~ N M / 2 M ! + l / 2 - 1 / 2 . M=0

As N -+ ~ , L -+ V/~zN/8.

(6)

The Case of Multiple Output Connections per Node, a > 1. In the case a > 1 the combinatorial approach gives rise to multiple summations. The resulting equations for total connectivity and path length are cumbersome and uninformative. Alternative approaches include the complete enumeration of all possible connections or a partial enumeration via Monte Carlo sampling procedure. In the course of obtaining just such samples on a high-speed computer, we realized that for large N, the sampling process could be accelerated using successive values of N[k], the expected number of nodes that are k steps from a given node. For small values of N these expectations are far from representative as can be seen from direct enumeration in some simple case. But for large N, these expectations do represent aggregate behavior. Thus, for large N, connectivity m a y be expressed b y the equation (Solomonoff and Rapoport, 1951) ? = Z N[k]/N

(7)

and the expected path length is given b y the equation L = X N[kJk/N.

(8)

CONNECTIVITY

I N RAR~DOM N E T W O R K S

299

In giving a physical interpretation to the notion of a r a n d o m network, o u t p u t connectivity m a y be viewed in a variety of ways. I n a neural network one m a y speak of a neuron giving rise to exactly a axons (constant o u t p u t connectivity) or of a neuron being linked to exactly a other neurons (constant outdegree). In the first ease Solomonoff and R a p o p o r t (1951) showed t h a t the N[/c] satisfy the following recursive equation: N[k] =

~ N [ j ] (1 - (1 - 1/N) aN[k-ll)

_AT -

(9)

j=0

A slight modification of this analysis gives rise to the following equation for the second case : N[]c] =

-

[j]

(1 - (1 -

a/nN[lc-1]))

(1O)

A s N increases, equation (9) or equation (10) m a y be written in the form N[/c] =

N-

N[j]

(1-exp [-aN[k-1]/N])

(11)

j=

The expected p a t h length, L, is related to the conneetance, a/N, as is shown by the following asymptotic result. T~EOR]~M 2 : I f a / N --~ C > 0 as N --+ 0% then 7 --+ 1 and L --~ 1 + exp [ - C]. Proof:

F r o m (11) N[1] = ( N - 1)(1 - e x p [ - a / N . 1]).

For suitably large N

Nil] ~ N ( 1 - e x p [ - e l ) ,

N[2] --~ N(1 - 1 + e x p [ - C])(1 - e x p [ - CN])

N exp [ - C], and N[k] = 0(l/N)

for

k>2.

Now L = ZkN[/c]/N Thus, N L = (1)(N)(1 - exp [ - C] + (2)(N) exp [ - C] + 0(l/N). L = 1 + e x p [ - C ] , as asserted.

(12)

If the connectance tends to zero when the number of nodes is large, t h e n the total connectiviby m a y be obtained from (11) b y a m e t h o d due to Solomonoff and Rapoport (1951) and L a n d a u (1952) 7 = 1-exp [-ay]. F

(13)

300

DEREK

F. S T U B B S A N D

P H I L L I P I. G O O D

A convenient expression relating the e x p e c t e d p a t h length a n d t h e o u t p u t connectivity is n o t readily derived. A l t e r n a t i v e l y , one m a y consider the radius. The radius is a m e a s u r e of t h e shortest p a t h to the f u r t h e s t node. A m o n g all the networks in #, the shortest p a t h occurs w h e n a nodes are one step away, a 2 nodes are two steps a w a y a n d so forth. T h e n u m b e r of nodes N is less t h a n or equal to 1 + a + a 2 + . . . + a ~ where k is t h e radius. N ~< (a k + l - 1 ) / ( a - 1)

(14)

k /> In [N(a- 1) + 1]/ln [a] - 1,

(1~)

or

(Wilson, 1972, p. 31). N o t e t h a t as _AT--~ c~, t h e n k - + l n N/ln a. TABLE I N e t w o r k c o n n e c t i v i t y as a f u n c t i o n of o u t p u t connectivity. E x a c t values for a = 1 ( E q u a t i o n 3). C o m p u t e r simulations for a > 1 Output Connectivit~

LO

i

.3660

2 3 4

.9743

5

.9900

.9999

.9999

10 20

Methods.

100~

Number of Nodes i000 10000

100000

7=l-exp[-ay]

.1221

.0393

,0125

.7519

,7924

.7960

.7969

.7968

.9251

.9394

.9394

.9404

.9405

.9803

,9802

.9802

.9802

.9930

.9930

.9930

.9930

I.

1.

.0440

1.

1.

1.

I.

W e w a n t e d to i n v e s t i g a t e the r a t e of convergence to our a s y m p t o t i c approximations. W e also w a n t e d to find t h e conditions u n d e r which the radius would come close to t h e e x p e c t e d p a t h length. W e p r o g r a m m e d a n I B M 165 c o m p u t e r to p e r f o r m a M o n t e Carlo e v a l u a t i o n of a r a n d o m n e t w o r k for given N and a. Specifically, the p r o g r a m g e n e r a t e d a succession of N x N matrices whose elements were selected b y a r a n d o m n u m b e r g e n e r a t o r subject to the restriction t h a t t h e s u m of t h e elements in each r o w was e q u a l to a. The network connectivity, e x p e c t e d p a t h length a n d v a r i a n c e of the p a t h length were determined for each such r a n d o m l y g e n e r a t e d n e t w o r k . T h e v a r i a n c e was used as a guide to t h e required n u m b e r of simulations. I n t h e case a = 1, (3) and (4) were used to generate the e x a c t values of t h e n e t w o r k c o n n e c t i v i t y a n d expected p a t h length. These values were checked against those o b t a i n e d via computer simulation. The results were v a l i d within one percent.

CONI~-ECTIVITY IX RANDOM

I~I-ETWORKS

301

Results. Table 1 provides the simulated network connectivity as a function of the output connectivity (for a = 1 the exact values are given). For a = 1 or even a = 2 the total connectivity is far from unity. In these cases there will be many isolated nodes or node sets within most networks. On the other hand, when the output connectivity is ten or greater, the total connectivity is close to unity even for large networks. The approximation of Solomonoff and Rapoport (1951), (13), is valid even for small networks of 100 or more nodes when the output connectivity is four or more. TABLE I I Expected path length and radius as a function of output connectivity. The exact expected path length obtained (viz. (4)) is compared with the asymptotic approximation for a = 1 (viz. (6)). The expected path length obtained via simulation is compared with the radius for a > 1 (viz. (15))

Output Cbnnectivity

k

N=IO --

~

k

5,605

Number o f Nodes N=IO0 N=IO00 - E L - E

1

1.300

1,982

6.267

2

2.441

2.459

5.576

5.658

3

2.095

1,771

4.022

3.827

4

1.839

1.477

3.359

5

1.684

1.307 1.000

19.152

19,152

N=IO000 ~ - -

62.000

62.666 12.288

8.822

8.967

12.156

.6,080

5,919

8.167

8.0146

3.117

4.985

4.776

6.641

6.436

2.978

2.724

4.386

4.154

5,813

5.584

2.254

1.955

3,241

2.954

4.232

3.954

20

1.840

1.520

2.693

2,289

3.393

3,057

40

i.669

1.242

2,162

1.866

2.846

2.490

1.

1.

1.905

1.498

2.359

1.998

1.

1.

1.905

1.333

I0

100 1000

Table I I provides the simulated expected path length and the radius as a function of the output connectivity. For a > 1 the expected path length is a decreasing function of the output connectivity. The singularity of the case of unit output connectivity is due to the m a n y isolated nodes; the expected path length is defined only with respect to those nodes t h a t are actually connected. The radius is a good approximation to the expected path length when the connectance, a/N, is small. For a = 1 the result of Theorem 1, (6), is also a good approximation if a/N is small. Table I I I gives the expected path length as a function of the connectance (a/N). For a connectance of one-tenth, the asymptotic approximation of

302

D E R E K F. S T U B B S A N D P H I L L I P I. GOOD

Theorem 2 is valid to three decimal places, the accuracy of the simulation, even for N = 1000. Discussion. The brain m a y be viewed as a network with the neurons as nodes and the axons and synapses as connections. I t has been shown (see review by Stubbs, 1975) t ha t the amount of information represented by the structure of the brain far exceeds the genetic information which originally specified that brain. Two, not necessarily incompatible, hypotheses stem from this observation. First, it may be t h a t the brain is an initially random network which is subsequently organized by experience (Stnbbs, 1975). Secondly, it m a y be that the genetic information available to specify the brain is used in an especially economical way (yon Foerster, 1967; H u n t and Stubbs, 1973). As a step toward elucidating these problems, we now derive a value for the output connectivity of the brain. TABLE III Expected path length as a function of the connectance, based on computer simulations for a > 1, and Equation 4 for a = 1

Connectance

10

Number o f Nodes i00 I000

I0000

l+exp[-a/N]

.01

5.605

3,241

2.359

1.990

.02

5.576

2.643

1.999

1.980

,10

1.330

2.254

1.905

1.905

1.905

.20

2,441

1.840

1.819

1,819

1.819

,40

1.839

1,669

1.670

1,670

1.670

As yet, a direct histological measure of the output connectivity for the human brain has not been made. The radius can be estimated. Assume t h a t the human brain is a randomly connected network. The number of nodes (neurons) is 101° (Warwick and Williams, 1973). The average number of steps between two neurons has been experimentally determined as 11 (Hunt and Stubbs, 1973). Using (15) for the radius, we can estimate the output connectivity as 10.5. We are thus able to specify for each neuron an expected number of neurons to which it should connect. This, biologically, m a y need only one gene. When yon Foerster (1967) computed the amount of information in the structure of the brain, he had to assume t h a t all values of a were possible. B y specifying the value of a, we considerably reduce the total information required. We may also use the value of the out pu t connectivity obtained to derive the value of the total connectivity of the brain. Using (13), we find t hat it is 0.99997302. Thus, nearly all of the 10 l° neurons of the hum an brain may be

CONNECTIVITY I N RANDOM N E T W O R K S

303

r e a c h e d d i r e c t l y or i n d i r e c t l y f r o m a n y o n e n e u r o n . H o w e v e r , s o m e 269,800 neurons would n o t be reached. E x p e r i m e n t a l c o n s i d e r a t i o n o f t h e s e v a l u e s a n d d i s c u s s i o n of t h e i r f u n c t i o n a l i m p l i c a t i o n s are g i v e n i n a f o r t h c o m i n g p u b l i c a t i o n ( S t u b b s , 1976). Mark J o h n s o n , J o h n R o b e r t s a n d R a y B a r n e s c o n t r i b u t e d t o t h i s i n v e s t i g a t i o n .

LITERATURE Alben, R. and P. Boutron. 1975. "Continuous Random Network Model for Amorphous Solid Water." Science, 187, 430-432. Anninos, P. A. and R. Elul. 1974. "Effect of Structure on Function in Model Nerve Nets." Biophys. J., 14, 9-19. Ashby, W. 1%., H. yon Foerster and C. C. Walker. 1962. "Instability of Pulse Activity in a Network with Threshold." Nature, 196, 561-562. Bell, R. J. and P. Dean. 1972. "Structure of Vitreous Silica. Validity of Random Network Theory." Philos. M a t . , 25, 1381-1398. Blumenson, L. E. 1970. "Random Walk and the Spread of Cancer." J. Theor. Biol., 27, 273-290. Cane, V. 1966. "A Note on the Size of Epidemics and the Number of People Hearing a Rumour." J. R. Star. Soe. Series B., 28, 487-490. Cor~eia, P. and M. T. Goveia. 1971, "Network Formation b y Random Cross Linking in Co-polymers." Rev. Pc. Quin., IS, 145-150. Daley, J. 1967. "Concerning the Spread of News in a Population of Individuals Who Never Forget." Bull. Math. Biophys., 29, 373-376. Doreian, P. 1974. "On the Connectivity of Social Networks." J. Math. Sociol., 3, 245-258. Farley, B. G. 1964. "The Use of Computer Technics in Neural Research." I n Neural Theory and Modeling, 1%. F. 1%eiss, ed., pp. 43-72. Stanford, California: Stanford U.P. Fitzhugh, H. S. 1963. "Some Considerations of Polystable Systems." I n Bionics Symposium. U.S.A.F. : Dayton, Ohio. Gardner, M. 1%. and W. 1%. Ashby. 1970. "Cormectance of Large Dynamic (Cybernetic) Systems: Critical Values for Stability." Nature, 228, 784. Griffith, J. S. 1963a. "On the Stability of Brain-like Structures." Biophys. J., S, 299-308. - - - - - - . 1963b. "A Field Theory of Neural Nets: I. Derivation of Field Equations." Bull. Math. Biophys,, 25, 111-120. .... 1965. "A Field Theory of Neural Nets : II. Properties of the Field Equations." Bull. Math. Biophys., 27, 187-195. Harmon, L. D. 1964. "Problems in Neural Modeling" I n Neural Theory and Modeling, 1%. F. Reiss, ed., pp. 9-30. Stanford, California: Stanford U.P. Hunt, J. N. and D. F. Stubbs. 1973. "Economy in the Brain." Lancet May 5, 1973, p. 997. Kauffman, S. A. 1969. "Metabolic Stability and Epigenesis in 1%andomly Constructed Nets." J. Theor. Biol., 22, 437-467. Landau, H. G. 1952. "On Some Problems of Random Nets." Bull. Math. Biophys., 14, 203-212. May, 1%. M. 1972. "Will a Large Complex System Be Stable ?" Nature, 2S8, 413-414. McMurtrie, 1%. E. 1975. "Determinants of Stability of Large Randomly Connected Systems." J. Theor. Biol., 50, 1-11. Newman, S. A. 1972. "A Source of Stability in Metabolic Networks." J. Theor. Biol., 35, 227-232. Park, D. J. M. 1975. "Local Stability in Metabolic Networks with Conserved Moieties and Steady State Subnetworks." J. Theoret. Biol., 49, 431-437. Rapoport, A. 1948. "Cycle Distribution in Random Nets." Bull. Math. Biophys., 10, 145-157.

304

DEREK F. STUBBS AND PHILLIP I. GOOD

Rapopert, A. 1953a. "Spread of Information Through a Population with Socio-structural Bias: I. Assumptions of Transitivity." Bull. Math. Biophys., 15, 523-533. • 1953b. "Spread of Information Through a Population with Socio-struetural Bias: II. Various Models with Partial Transitivity." Bull• Math. Biophys., 15, 535-547. . 1954. "Spread of Information Through a Population with Socio-structural Bias: III. Suggested Experimental Procedures." Bull. Math. Biophys., 16, 75-81. Rashevshy, N. 1960. Mathematical Biophysics, pp. 230-241. New York: Dover. Roberts, A. 1974. "Stability of a Feasible R a n d o m Ecosystem." Nature, 251, 607-608. Schu]tz, J. V. and L. J. Hubert. 1973. " D a t a Analysis and the Connectivity of Random Graphs." J. Math. Psychol., 1O, 421-428. Shimbel, A. 1948. " A n Analysis of Theoretical Systems of Differentiating Nervous Tissue." Bull. Math• Biophys., 10, 131-143. Siljak, D. D. 1974. "Connective Stability of Complex Ecosystems." Na$ure~ 249, 280. Solomonoff, R. and A. Rapoport. 1951. "Connectivity of R a n d o m Nets." Bull. Math. Biophys., 13, 107-117. Somorjai, R. L. and Goswami, D. N. 1972. "Relationship Between Stability and Connectedness in Non-Linear Systems." Nature, 236, 466• Stubbs, D. F. 1975. "Perceptual Learning Machines and the Brain•" Stochastics, 1, 301-314. • 1976. "The Connectivity of the Brain•" Paper in preparation. von Foerster, H. 1967. "Memory Without Record." I n The Anatomy of Memory, D. P. Kimble, ed., pp. 388-433. Palo Alto, California: Science and Behavior Books, Inc. Walker, C. C. and Ashby, W. 1%. 1966. "On Temporal Characteristics of Behavior in Certain Complex Systems." KybernetiIc, 3, 100-108. Warwick, R. and P. Williams. 1973. Anatomy. London: Longman. Wiener, N. and A. Rosenblueth. 1946. "The Mathematical Formulation of Conduction of Impulses in a Network of Connected Excitable Elements, Specifically in Cardiac Muscle." Arch• Inst. Cardiolog. Mex., X V I , 205-265. Wilson, R. J. 1972. Introduction to Graph Theory. Glasgow: Bell and Bain.

RECEIVED 7-28-75