IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
[19] J. Besag, “Spatial interaction and the statistical analysis of lattice systems,” J. Roy. Stat. Soc. B, vol. 36, no. 2, pp. 192–326, 1974. [20] W. Rudin, Real and Complex Analysis. New York: McGraw-Hill, 1987, pp. 62–65. [21] K. Blekas, A. Likas, N. P. Galatsanos, and I. E. Lagaris, “A spatially constrained mixture model for image segmentation,” IEEE Trans. Neural Netw., vol. 16, no. 2, pp. 494–498, Feb. 2005. [22] [Online]. Available: http://infoman.teikav.edu.gr/∼stkrini/pages/develop/ FLICM/FLICM.html [23] [Online]. Available: http://web.mac.com/soteri0s/Sotirios_Chatzis/ Software.html [24] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in Proc. 8th IEEE Int. Conf. Comput. Vis., Jun. 2001, pp. 416–423. [25] R. Unnikrishnan and M. Hebert, “Measures of similarity,” in Proc. IEEE Workshop Comput. Vis. Appl. Conf., May 2005, pp. 394–400. [26] R. Unnikrishnan, C. Pantofaru, and M. Hebert, “A measure for objective evaluation of image segmentation algorithms,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2005, pp. 34–41.
Hyperbolic Hopfield Neural Networks Masaki Kobayashi
Abstract— In recent years, several neural networks using Clifford algebra have been studied. Clifford algebra is also called geometric algebra. Complex-valued Hopfield neural networks (CHNNs) are the most popular neural networks using Clifford algebra. The aim of this brief is to construct hyperbolic HNNs (HHNNs) as an analog of CHNNs. Hyperbolic algebra is a Clifford algebra based on Lorentzian geometry. In this brief, a hyperbolic neuron is defined in a manner analogous to a phasor neuron, which is a typical complex-valued neuron model. HHNNs share common concepts with CHNNs, such as the angle and energy. However, HHNNs and CHNNs are different in several aspects. The states of hyperbolic neurons do not form a circle, and, therefore, the start and end states are not identical. In the quantized version, unlike complex-valued neurons, hyperbolic neurons have an infinite number of states. Index Terms— Clifford algebra, complex-valued neural networks, Hopfield neural networks (HNNs), hyperbolic algebra.
I. I NTRODUCTION Complex-valued neural networks, which are an extension of ordinary neural networks, have been studied by many researchers. In recent years, further extensions based on Clifford algebra have been proposed. Clifford algebra, also called geometric algebra, encompasses the fields of complex numbers and quaternions, and hyperbolic algebra [1]. Clifford algebras produce inherent geometric properties. For example, the multiplication of a complex number is an extension and rotation operation in the complex plane. In the field of feed-forward complex-valued neural networks, researchers have mainly concentrated on real– Manuscript received February 9, 2012; revised October 11, 2012; accepted November 23, 2012. Date of publication December 20, 2012; date of current version January 11, 2013. The author is with the Interdisciplinary Graduate School of Medicine and Engineering, University of Yamanashi, Yamanashi 400-8511, Japan (e-mail: [email protected]). Digital Object Identifier 10.1109/TNNLS.2012.2230450
335
imaginary and amplitude–phase type complex-valued neural networks. The backpropagation learning rule for real– imaginary type complex-valued neural networks was first proposed by Benvenuto and Piazza [2] and Nitta [3], [4]. The backpropagation learning rule for amplitude–phase type complex-valued neural networks was proposed by Hirose [5]. Further extensions to Clifford neural networks have been proposed. Nitta [6] proposed a neural network model using quaternions that realizes partial rotation in a 4-D space [7]. Another proposed neural network model with quaternions realizes full rotation in a 3-D space [8]. In Clifford algebra, hyperbolic algebra can process information in non-Euclidean space. Ritter [9] proposed hyperbolic self-organizing map. Ontrup and Ritter [10], [11] applied it to text categorization and semantic navigation. Buchholtz and Sommer [12] and Nitta and Buchholz [13] studied hyperbolic neural networks. Hopfield [14] proposed a recurrent neural network model referred to as the Hopfield neural networks (HNNs). HNNs have been extended to complex-valued Hopfield neural networks (CHNNs). Phasor neural networks are the simplest and most popular models of CHNNs [15]–[17]. In particular, quantized phasor neural networks, also referred to as multivalued neural networks, have been applied to multilevel data such as gray-scale images [18]– [20]. Several extensions to Clifford HNNs have been attempted. Isokawa et al. [21], [22] studied HNNs using quaternion. Kuroe [23] constructed hyperbolic HNNs (HHNNs) as an analog of real–imaginary type CHNNs. However, it is difficult to construct HHNNs as an analog of phase–amplitude type CHNNs. The HNN model with commutative quaternions proposed by Isokawa et al. [24] is incomplete because it requires HHNNs as an analog of phase–amplitude type HHNNs. Thus, HHNNs are necessary to study HNNs with higher dimensional Clifford algebra. The aim of the present brief is to construct an HHNN model as an analog of a phasor CHNN. The proposed HHNN shares common concepts with a CHNN, such as the angle and energy. However, HHNNs and CHNNs are different in many aspects. The states of hyperbolic neurons do not form a circle, and, therefore, the start and end states are not identical. In the quantized version, unlike complex-valued neurons, hyperbolic neurons can have an infinite number of states. The rest of this brief is organized as follows. Section II provides an introduction to CHNNs, which is used as an outline for constructing HHNNs. Section III describes hyperbolic algebra, which is necessary for constructing an HHNN. Then, we construct an HHNN in Section IV. Finally, Section V concludes this brief. II. CHNN S In this section, we briefly describe the conventional theory behind CHNNs [16]–[18]. The aim of this brief is to construct HHNNs as an analog of CHNNs. This section provides an outline for constructing HHNNs given in Section IV.
2162–237X/$31.00 © 2012 IEEE
336
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
imaginary
follows: E =−
input output
−1
1
It is easy to prove E = E. Therefore, E is a real number. When a neuron is updated, the energy always decreases in the wider sense. For state vector z = (z 1 , z 2 , . . . , z N ), consider the rotated state eiθ z = (eiθ z 1 , eiθ z 2 , . . . , eiθ z N ). Then, the energy of the rotated state is as follows: 1 iθ E =− e z k wkj eiθ z j (5) 2
real
−i
Fig. 1. Activation function of complex-valued neurons. The output is the intersection of the complex unit circle and the line passing through the origin and input.
(1)
If z is expressed in the polar form, r exp(i θ ), then f (z) = exp(i θ ). Fig. 1 illustrates the activation function f (z). From a geometric point of view, f (z) is the complex number on the complex unit circle closest to z. Let z r and z be the real part and complex conjugate of z, respectively. Then, f (z) is the complex number c such that c maximizes (cz)r , where c runs over the complex unit circle. The set of complex-valued states of a neuron is cyclic. Suppose that the target data range is [0, 1]. We typically map the target values x to the state exp(i (2π x)) of complex-valued neurons. Then, both 0 and 1 are mapped to the same state. However, the two ends should be mapped to different states. This is a problem caused by complex-valued neurons. B. CHNNs Each neuron of a CHNN is connected to all other neurons. Let the connection weight from neuron j to neuron k be denoted by wkj . Then, the following relation is required to ensure that a CHNN reaches a stable state: wkj = w j k .
(2)
We denote the state of neuron j as z j . Then, the weighted sum input Ik to neuron k is expressed as follows: Ik = wkj z j . (3) j =k
C. Energy Function Let the state of a CHNN be z = (z 1 , z 2 , . . . , z N ), where N is the number of neurons. Then, the energy E is defined as
j =k
k
j =k
k
j =k
(6)
1 z k wkj z j . 2
(7)
A. Complex-Valued Neurons The input and output of complex-valued neurons are complex numbers. For an input z, the output of a complex-valued neuron is determined by an activation function. In this brief, for a complex number z = x + i y, where i is the imaginary unit, we define an activation function f (z) as follows:
k
1 −iθ =− e z k wkj eiθ z j 2 =−
x + iy z = . |z| x 2 + y2
(4)
j =k
k
i
f (z) =
1 z k wkj z j . 2
Therefore, the energies of a pattern and its rotated patterns are equal. This fact is referred to as rotation invariance [25]. D. Complex-Valued Hebbian Learning Rule The complex-valued Hebbian learning rule is the simplest learning method for a CHNN. Let the pth training pattern p p p vector be denoted by (c1 , c2 , . . . , c N ) ( p = 1, 2, . . . , P), where P is the number of training patterns. Then, the complexvalued Hebbian learning rule produces the following connection weights: p p ck c j . (8) wkj = p
When the qth training pattern is applied to a CHNN, the weighted sum input Ik to neuron k is as follows: p p q Ik = ck c j c j (9) j =k p q
= (N − 1)ck +
p p q
ck c j c j .
(10)
j =k p =q
If the second term of the right-hand side is small enough, we can conclude that the qth training pattern is stable. The second term is referred to as the crosstalk term. E. Quantized CHNNs The quantized version of a CHNN is often applied. The CHNN and the quantized CHNN have different activation functions. Let K be a positive integer greater than two. The activation function f (z) of a quantized complex-valued neuron is defined as follows: ⎧ 1 0 ≤ arg(z) < θ K ⎪ ⎪ ⎪ ⎪ exp(i (2θ )) θ K ≤ arg(z) < 3θ K ⎪ K ⎪ ⎪ ⎪ exp(i (4θ )) 3θ ⎪ K K ≤ arg(z) < 5θ K ⎪ ⎪ ⎨ .. .. . (11) f (z) = . ⎪ )) (2K − 3)θ K ≤ arg(z) exp(i (2(K −1)θ ⎪ K ⎪ ⎪ ⎪ < (2K − 1)θ K ⎪ ⎪ ⎪ ⎪ 1 (2K − 1)θ K ≤ arg(z) ⎪ ⎪ ⎩ < 2π
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
337
unipotent
imaginary input
u
i output −1
−1
real
1
1
real
−u
−i
Fig. 2. Complex-valued activation function (K = 8). Dotted lines represent the decision boundaries.
where θ K = π/K . Hence, the number of complex-valued states of a neuron is K . Fig. 2 illustrates the activation function of quantized complex-valued neurons in the case of K = 8. The dotted lines represent the decision boundaries. Because quantized complex-valued neurons have multiple states, quantized CHNNs are often applied to multilevel data such as gray-scale images. Before using a CHNN, we have to select and fix the value of K . If a new data element exceeds K , we cannot apply the data to the CHNN. Thus, quantized CHNNs lack flexibility.
Fig. 3. Domain D = {x + uy; x > 0, x 2 > y 2 } and curve S = {x + uy ∈ D; x 2 − y 2 = 1}.
follows: D = {z = x + uy ∈ H; x > 0, x 2 > y 2 }.
(16)
Domain D is the right hyperbolic quadrant of the hyperbolic plane. The following subset S of D is of particular importance: S = {z ∈ D; |z| = 1}.
(17)
Fig. 3 illustrates domain D and the curve representing S on the hyperbolic plane. For a hyperbolic number ut, where t is a real number, we define the exponential function exp ut as follows:
III. H YPERBOLIC N UMBERS
exp ut = cosh t + u sinh t.
In this section, we describe hyperbolic numbers. A unipotent number u is one that simultaneously satisfies the algebraic properties u = ±1 and u 2 = 1. Hyperbolic numbers are written in the form x + uy, where x and y are real numbers. The set of all hyperbolic numbers is denoted by H. Hyperbolic numbers are 2-D vectors over the field of real numbers, and are identified by points in the 2-D real plane. This 2-D real plane is referred to as the hyperbolic plane. Addition and multiplication in H are defined as follows:
Proposition 3.1: S = {exp ut; t ∈ R}. Proof: By corresponding z = x + uy ∈ S with t = log(x + y), we complete the proof. Any element z = x + uy ∈ D is uniquely represented in the hyperbolic polar form as
(x + uy) + (x + y u) = (x + x ) + u(y + y )
(12)
(x + uy)(x + uy ) = (x x + yy ) + u(x y + x y). (13) Multiplication is clearly commutative. For a hyperbolic number z = x + uy, the real numbers x and y are referred to as the real and unipotent parts of z, respectively. In this brief, the real part of z is denoted by z r . The hyperbolic conjugate z and the hyperbolic modulus |z| of a hyperbolic number z = x + uy are defined as follows: z = x − uy ! |z| = |zz| = |x 2 − y 2 |.
(14) (15)
It is easy to prove that z 1 z 2 = z 1 z 2 for any hyperbolic numbers z 1 and z 2 . In this brief, we do not need to consider the entire range of hyperbolic numbers. We require only the domain D defined as
(18)
z = r exp(ut) (19) ! (20) r = |z| = x 2 − y 2 y t = tanh−1 . (21) x Parameter t is referred to as the hyperbolic angle. Proposition 3.2: For hyperbolic numbers z 1 = r1 exp ut1 and z 2 = r2 exp ut2 ∈ D, the following equation holds: z 1 z 2 = r1r2 exp u(t1 + t2 ).
(22)
Proof: z 1 z 2 = r1r2 exp ut1 exp ut2
(23)
= r1r2 (cosh t1 + u sinh t1 )(cosh t2 + u sinh t2 ) (24) = r1r2 ((cosh t1 cosh t2 + sinh t1 sinh t2 ) +u(cosh t1 sinh t2 + sinh t1 cosh t2 )) = r1r2 (cosh(t1 + t2 ) + u sinh(t1 + t2 ))
(25) (26)
= r1r2 (exp u(t1 + t2 )).
(27)
338
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
unipotent
unipotent
u input output
real
−1
1
real
−u
Fig. 4. Hyperbolic rotation. Through multiplication by exp ut, a point in D moves while following the hyperbolic modulus.
Fig. 5. Hyperbolic activation function. The output is the intersection of curve S and the line passing through the origin and the input.
IV. HHNN S Multiplying by exp ut follows the hyperbolic modulus and is referred to as hyperbolic rotation (Fig. 4). Proposition 3.3: D is closed under addition. Proof: Trivial. Proposition 3.4: D is a multiplicative group. Proof: From Proposition 3.2, D is closed under multiplication. Hyperbolic algebra includes the associative law. It is clear that unity (1) belongs to D. The inverse of z = r exp ut ∈ D is r −1 exp(−ut). Therefore, D is a multiplicative group. Proposition 3.5: For any z ∈ S, z belongs to S, and is the inverse of z. Proof: For z = exp ut, we show that z = exp u(−t). z = cosh t − u sinh t = cosh(−t) + u sinh(−t) = exp u(−t).
(28) (29) (30)
Hence, we proved that z ∈ S. From Proposition 3.2, we get the following equation: zz = exp ut exp u(−t) = exp u(t − t) = exp u0 = 1.
A. Hyperbolic Neuron In this subsection, we define hyperbolic neurons. Inputs of hyperbolic neurons are hyperbolic numbers in D. Outputs of hyperbolic neurons are hyperbolic numbers in S. For an input z, a hyperbolic neuron with activation function g(z) produces the output g(z). For z = x + uy ∈ D, we define the activation function g(z) as follows: g(z) =
x + uy z = . |z| x 2 − y2
(34)
We call this activation function g(z) the hyperbolic activation function. When we represent z = r exp ut in hyperbolic polar form, we can write g(z) = exp ut. From a geometrical point of view, g(z) is the intersection of curve S and the line passing through the origin and point z on the hyperbolic plane. Fig. 5 illustrates the relation between z and g(z). The following proposition is of particular importance. Proposition 4.1: For z ∈ D, g(z) is equal to c ∈ S and c minimizes (cz)r . Proof: We write z and c in hyperbolic polar form as follows:
(31)
z = r exp utz
(35)
(32) (33)
c = exp utc .
(36)
Therefore, z is the inverse of z. Proposition 3.6: S is a multiplicative group. Proof: We prove that S is closed under multiplication. For any z 1 = exp ut1 and z 2 = exp ut2 ∈ S, equation z 1 z 2 = exp u(t1 + t2 ) holds. Thus we proved that z 1 z 2 ∈ S. Hyperbolic algebra includes the associative law. It is clear that unity (1) belongs to D. From Proposition 3.5, the inverse of z exists for any z ∈ S. This concludes the proof. We can find some correspondences between the complex plane and domain D in the hyperbolic plane. Domain S, which is the set of points with modulus 1, corresponds to the complex unit circle. The hyperbolic polar form and hyperbolic angle correspond to the polar form and phase of a complex plane, respectively.
Then, we get g(z) = exp utz . Moreover (cz)r = (r exp u(tz − tc ))r = r cosh(tz − tc ).
(37) (38)
If and only if tz = tc , (cz)r is minimized. Therefore, c = exp utz minimizes (cz)r , which concludes the proof. Suppose that the target data belongs in the range [0, 1]. In the case of a complex-valued neuron, both 0 and 1 are typically mapped to the same state. In the case of a hyperbolic neuron, if we map the target value x to exp u(2x − 1)A, where A is an arbitrary positive number, 0 and 1 are mapped to the most different states exp u(−A) and exp u A, respectively. Then, the area used by the states of the neuron spans from exp u(−A) to exp u A. Thus, for large values of A we can use a wider range.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
B. HHNNs In this subsection, we construct a HHNN. First, we define connection weights. Every neuron of the HHNN is connected to all other neurons. We denote the connection weight from neuron j to neuron k by wkj . The connection weights must satisfy the following conditions: 1) wkj ∈ D; 2) wkj = w j k . We denote the output of neuron k by z k . We define the weighted sum input Ik to neuron k as follows: wkj z j . (39) Ik = j =k
Thus, Ik belongs to D. Neuron k receives the weighted sum input Ik and produces g(Ik ).
Let the state of HHNN be z = (z 1 , z 2 , . . . , z N ), where N is the number of neurons. We define the energy function E of HHNN as follows: 1 E = z k wkj z j . (40) 2 k
j =k
It is easy to prove E = E. Therefore, E is a real number. Suppose that the state z l of neuron l changes to z l . Then, the energy gap E is as follows: ⎞ ⎛ 1 ⎝ z k wkl z l + z l wl j z j ⎠ E = 2 k =l j =l ⎞ ⎛ 1 − ⎝ z k wkl z l + z l wl j z j ⎠ (41) 2 k =l j =l ⎞r ⎛ ⎞r ⎛ z l wl j z j ⎠ − ⎝ z l wl j z j ⎠ (42) =⎝ j =l
= ⎝z l
⎞r
⎛
j =l
wl j z j ⎠ − ⎝z l
j =l
⎞r wl j z j ⎠
(43)
j =l
= (z l Il )r − (z l Il )r .
(44)
From Proposition 4.1, z l minimizes (z Il )r . Hence, E is nonpositive, and E does not increase. Because j =k z k wkj z j belongs to D, the energy is definitely positive. Therefore, the energy always converges. For state vector z = (z 1 , z 2 , . . . , z N ), consider the hyperbolic rotated state eut z = (eut z 1 , eut z 2 , . . . , eut z N ). Then, the energy of the hyperbolic rotated state is as follows: 1 ut e z k wkj eut z j (45) E = 2 k
j =k
k
j =k
k
j =k
1 u(−t ) = e z k wkj eut z j 2 =
Therefore, the energies of a pattern and its hyperbolic rotated patterns are equal. This property is called hyperbolic rotation invariance. D. Hyperbolic Hebbian Learning Rule Denote the state of neuron j of the pth training pattern by p c j . Then, we define the connection weights as follows: p p ck c j . (48) wkj = p
We call this learning rule the hyperbolic Hebbian learning rule. p From Propositions 3.3 and 3.4, and the fact that c j ∈ D, wkj ∈ D is true. When the qth training pattern is given, the weighted sum input Ik to neuron k is as follows: p p q ck c j c j (49) Ik = j =k p
C. Energy Function
⎛
339
1 z k wkj z j . 2
(46) (47)
q
= (N − 1)ck +
p p q
ck c j c j .
(50)
j =k p =q
If the crosstalk terms are small enough, we can conclude that the qth training pattern is stable as in the case of CHNNs. p p q However, the crosstalk term is large. From ck c j c j ∈ S, the real part of the crosstalk term is greater than (N − 1)(P − 1). Therefore, hyperbolic Hebbian learning rule cannot deal with multiple patterns. E. Quantized HHNNs In this section, we consider quantizing the HHNN. We take a positive number T and call it a hyperbolic resolution factor. We take an infinite number of discrete points exp u(nT ) on S, where n is an integer, and define the set {exp u(nT )}∞ n=−∞ to be the states of the quantized hyperbolic neurons. Next, we have to create a quantized version of the activation function to decrease the energy function. For a hyperbolic number z to decrease the energy, it suffices to set the activation function to exp u(nT ), where n minimizes (z exp u(−nT ))r , because then expression (44) is nonpositive. Next, we determine the decision boundary between the states exp u(nT ) and exp u(n + 1)T . A point z on the decision boundary satisfies (z exp u(−nT ))r = (z exp u(−n − 1)T )r . By expressing z = r exp ut, we obtain the following equations: (r exp u(t − nT ))r = (r exp u(t − (n + 1)T ))r
(51)
cosh(t − nT ) = cosh(t − (n + 1)T ) (52) t − nT = (n + 1)T − t (53) 1 (54) t = (n + )T. 2 Thus, we obtain the decision boundary z = a exp u(n +1/2)T , where a is a positive parameter. For z = r exp ut ∈ D, if (n − 1/2)T < t < (n + 1/2)T , then f (z) = exp u(nT ). In the case of complex-valued neurons, we have to fix the number of states K . If a new data element exceeds the fixed range, we cannot apply the data to the CHNN. Because hyperbolic neurons have an infinite number of states, the range of data is not restricted.
340
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
unipotent
geometric properties. In the future, we will discuss our model from the geometric point of view to find applicable targets, such as text categorization and semantic navigation, which need information processing in non-Euclidean space.
u input output −1
1
R EFERENCES real
−u
Fig. 6. Quantized hyperbolic activation function. Dotted lines are the decision boundaries.
If we want to restrict the number of states to be finite, we can take a finite number of sequential integers n. Fig. 6 illustrates the states of a quantized hyperbolic neuron in the case of five states. The range of n is from −2 to 2. The dotted lines are the decision boundaries. V. C ONCLUSION In this brief, we constructed an HHNN as an analog of a CHNN. HHNNs share many common concepts with CHNNs such as the angle, rotation invariance, energy function, and so on. On the other hand, HHNNs posses some properties that are better than those of CHNNs. 1) The set of complex-valued states of a neuron forms a circle, and, thus, it is hard to properly map the target data to CHNN; on the other hand, the set of hyperbolic states of a neuron is homeomorphic to the real number line, and, thus, it is easy to naturally map the target data to a HHNN. 2) Unlike quantized complex-valued neurons, quantized hyperbolic neurons have an infinite number of states; CHNNs can handle only a fixed range of data, while HHNNs do not restrict the range of data. If the range of the target data is difficult to predict beforehand, HHNN may be suitable. In our construction, hyperbolic neurons receive only hyperbolic numbers belonging to the right quadrant of the hyperbolic plane. Since hyperbolic Hebbian learning rule does not work well, the performance of HHNNs is worse than that of CHNNs at present, though they have some better properties. So we have to develop practical learning methods. Projection rule and gradient descent learning rule for CHNN have been proposed [19], [26]– [28]. We should develop those for HHNN. Then, it is possible that connection weights do not belong to the right quadrant. In this case, the hyperbolic activation function should be extended to the entire hyperbolic plane. HHNNs are expected to be applicable when constructing higher dimensional Clifford HNNs. For example, Isokawa et al. [24] attempted to construct an HNN with commutative quaternions. Their proposed model requires the use of a HHNN, but their work is still incomplete. In this brief, we mainly discussed our model from the algebraic point of view. Hyperbolic algebra has excellent
[1] S. Buchholz, “A theory of neural computation with Clifford algebra,” Ph.D. dissertation, Dept. Comput. Sci., Univ. Kiel, Kiel, Germany, 2005. [2] N. Benvenuto and F. Piazza, “Multi-valued and universal binary neurons: Theory, learning and application,” IEEE Trans. Signal Process., vol. 40, no. 4, pp. 967–969, May 1992. [3] T. Nitta, “Complex numbered version of the back-propagation algorithm,” in Proc. INNS World Congr. Neural Netw., vol. 3, pp. 576–579, 1993. [4] T. Nitta, “An extension of the back-propagation algorithm to complex numbers,” Neural Netw., vol. 10, no. 8, pp. 1391–1415, 1997. [5] A. Hirose, “Continuous complex-valued back-propagation learning,” Electron. Lett., vol. 28, no. 20, pp. 1854–1855, 1992. [6] T. Nitta, “An extension of the back-propagation algorithm to quaternions,” in Proc. Int. Conf. Neural Inf. Process., vol. 1. 1996, pp. 247–250. [7] M. Kobayashi, J. Muramatsu, and H. Yamazaki, “Construction of high-dimensional neural networks by linear connections of matrices,” Electron. Commun. Japan III, vol. 86, no. 3, pp. 38–45, 2003. [8] P. Arena, L. Fortuna, G. Muscato, and M. G. Xibilia, Neural Networks in Multidirectional Domains. London, U.K.: Springer, 1998. [9] H. Ritter, “Self-organizing maps on non-euclidean spaces,” Kohonen Maps, E. Oja, S. Kaski, Eds. Amsterdam, The Netherlands: Elsevier, pp. 97–110, 1999. [10] J. Ontrup and H. Ritter, “Hyperbolic self-organizing maps for semantic navigation,” Advances in Neural Information Processing Systems, vol. 14. Cambridge, MA: MIT Press, 2001, pp. 1417–1424. [11] J. Ontrup and H. Ritter, “Text categorization and semantic browsing with self-organizing maps on non-euclidean spaces,” in Proc. 5th Eur. Conf. Principles Pract. Knowl. Discovery Databases, 2001, pp. 338–349. [12] S. Buchholz and G. Sommer, “A hyperbolic multilayer perceptron,” in Proc. IEEE-INNS-ENNS Int. Joint Conf. Neural Netw., vol. 2. Feb. 2000, pp. 129–133. [13] T. Nitta and S. Buchholz, “On the decision boundaries of hyperbolic neurons,” in Proc. Int. Joint Conf. Neural Netw., 2008, pp. 2973–2979. [14] J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities,” Proc. Nat. Acad. Sci. United States Amer., vol. 79, no. 8, pp. 2554–2558, 1982. [15] S. Jankowski, A. Lozowski, and J. M. Zurada, “Complex-valued multistate neural associative memory,” IEEE Trans. Neural Netw., vol. 7, no. 6, pp. 1491–1496, Nov. 1996. [16] A. J. Noest, “Phasor neural networks,” in Neural Information Processing Systems, D. Z. Anderson, Ed. New York: AIP, 1988, pp. 584–591. [17] A. J. Noest, “Discrete-state phasor neural networks,” Phys. Rev. A, vol. 38, no. 4, pp. 2196–2199, 1988. [18] I. N. Aizenberg, N. N. Aizenberg, and J. Vandewalle, Multi-Valued and Universal Binary Neurons: Theory, Learning and Application. Boston, MA: Kluwer Academic, 2000. [19] H. Aoki, “A complex-valued neuron to transform gray level images to phase information,” in Proc. Int. Conf. Neural Inf. Process., vol. 3, pp. 1084–1088, 2002. [20] M. K. Muezzinoglu, C. Guzelis, and J. M. Zurada, “A new design method for the complex-valued multistate Hopfield associative memory,” IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 891–899, Jul. 2003. [21] T. Isokawa, H. Nishimura, N. Kamiura, and N. Matsui, “Associative memory in quaternionic Hopfield neural network,” Int. J. Neural Syst., vol. 18, no. 2, pp. 135–145, 2008. [22] T. Isokawa, H. Nishimura, and N. Matsui, “An iterative learning scheme for multistate complex-valued and quaternionic Hopfield neural networks,” in Proc. Int. Joint Conf. Neural Netw., pp. 1365–1371, 2009. [23] Y. Kuroe, S. Tanigawa, and H. Iima, “Models of Hopfield-type Clifford neural networks and their energy functions - hyperbolic and dual valued networks,” in ICONIP Part I (Lecture Notes in Computer Science), B. L. Lu, L. Zhang, and J. Kwok, Eds. Berlin, Germany: SpringerVerlag, 2011, pp. 560–569.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
[24] T. Isokawa, H. Nishimura, and N. Matsui, “Commutative quaternion and multistate hopfield neural networks,” in Proc. Int. Joint Conf. Neural Netw., pp. 1281–1286, 2010. [25] R. S. Zemel, C. K. I. Williams, and M. C. Mozer, “Lending direction to neural networks,” Neural Netw., vol. 8, no. 4, pp. 503–512, 1995. [26] M. Kitahara and M. Kobayashi, “Projection rules for complex-valued associative memory with large constant terms,” Nonlinear Theory Appl., vol. 3, no. 3, pp. 426–435, 2012.
341
[27] M. Kobayashi, H. Yamada, and M. Kitahara, “Noise robust gradient descent learning for complex-valued associative memory,” IEICE Trans. Fundamentals Electron., Commun. Comput. Sci., vol. E94-A, no. 8, pp. 1756–1759, 2011. [28] D. L. Lee, “Improving the capacity of complex-valued neural networks with a modified gradient descent learning rule,” IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 439–443, Mar. 2001.
[19] J. Besag, “Spatial interaction and the statistical analysis of lattice systems,” J. Roy. Stat. Soc. B, vol. 36, no. 2, pp. 192–326, 1974. [20] W. Rudin, Real and Complex Analysis. New York: McGraw-Hill, 1987, pp. 62–65. [21] K. Blekas, A. Likas, N. P. Galatsanos, and I. E. Lagaris, “A spatially constrained mixture model for image segmentation,” IEEE Trans. Neural Netw., vol. 16, no. 2, pp. 494–498, Feb. 2005. [22] [Online]. Available: http://infoman.teikav.edu.gr/∼stkrini/pages/develop/ FLICM/FLICM.html [23] [Online]. Available: http://web.mac.com/soteri0s/Sotirios_Chatzis/ Software.html [24] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in Proc. 8th IEEE Int. Conf. Comput. Vis., Jun. 2001, pp. 416–423. [25] R. Unnikrishnan and M. Hebert, “Measures of similarity,” in Proc. IEEE Workshop Comput. Vis. Appl. Conf., May 2005, pp. 394–400. [26] R. Unnikrishnan, C. Pantofaru, and M. Hebert, “A measure for objective evaluation of image segmentation algorithms,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2005, pp. 34–41.
Hyperbolic Hopfield Neural Networks Masaki Kobayashi
Abstract— In recent years, several neural networks using Clifford algebra have been studied. Clifford algebra is also called geometric algebra. Complex-valued Hopfield neural networks (CHNNs) are the most popular neural networks using Clifford algebra. The aim of this brief is to construct hyperbolic HNNs (HHNNs) as an analog of CHNNs. Hyperbolic algebra is a Clifford algebra based on Lorentzian geometry. In this brief, a hyperbolic neuron is defined in a manner analogous to a phasor neuron, which is a typical complex-valued neuron model. HHNNs share common concepts with CHNNs, such as the angle and energy. However, HHNNs and CHNNs are different in several aspects. The states of hyperbolic neurons do not form a circle, and, therefore, the start and end states are not identical. In the quantized version, unlike complex-valued neurons, hyperbolic neurons have an infinite number of states. Index Terms— Clifford algebra, complex-valued neural networks, Hopfield neural networks (HNNs), hyperbolic algebra.
I. I NTRODUCTION Complex-valued neural networks, which are an extension of ordinary neural networks, have been studied by many researchers. In recent years, further extensions based on Clifford algebra have been proposed. Clifford algebra, also called geometric algebra, encompasses the fields of complex numbers and quaternions, and hyperbolic algebra [1]. Clifford algebras produce inherent geometric properties. For example, the multiplication of a complex number is an extension and rotation operation in the complex plane. In the field of feed-forward complex-valued neural networks, researchers have mainly concentrated on real– Manuscript received February 9, 2012; revised October 11, 2012; accepted November 23, 2012. Date of publication December 20, 2012; date of current version January 11, 2013. The author is with the Interdisciplinary Graduate School of Medicine and Engineering, University of Yamanashi, Yamanashi 400-8511, Japan (e-mail: [email protected]). Digital Object Identifier 10.1109/TNNLS.2012.2230450
335
imaginary and amplitude–phase type complex-valued neural networks. The backpropagation learning rule for real– imaginary type complex-valued neural networks was first proposed by Benvenuto and Piazza [2] and Nitta [3], [4]. The backpropagation learning rule for amplitude–phase type complex-valued neural networks was proposed by Hirose [5]. Further extensions to Clifford neural networks have been proposed. Nitta [6] proposed a neural network model using quaternions that realizes partial rotation in a 4-D space [7]. Another proposed neural network model with quaternions realizes full rotation in a 3-D space [8]. In Clifford algebra, hyperbolic algebra can process information in non-Euclidean space. Ritter [9] proposed hyperbolic self-organizing map. Ontrup and Ritter [10], [11] applied it to text categorization and semantic navigation. Buchholtz and Sommer [12] and Nitta and Buchholz [13] studied hyperbolic neural networks. Hopfield [14] proposed a recurrent neural network model referred to as the Hopfield neural networks (HNNs). HNNs have been extended to complex-valued Hopfield neural networks (CHNNs). Phasor neural networks are the simplest and most popular models of CHNNs [15]–[17]. In particular, quantized phasor neural networks, also referred to as multivalued neural networks, have been applied to multilevel data such as gray-scale images [18]– [20]. Several extensions to Clifford HNNs have been attempted. Isokawa et al. [21], [22] studied HNNs using quaternion. Kuroe [23] constructed hyperbolic HNNs (HHNNs) as an analog of real–imaginary type CHNNs. However, it is difficult to construct HHNNs as an analog of phase–amplitude type CHNNs. The HNN model with commutative quaternions proposed by Isokawa et al. [24] is incomplete because it requires HHNNs as an analog of phase–amplitude type HHNNs. Thus, HHNNs are necessary to study HNNs with higher dimensional Clifford algebra. The aim of the present brief is to construct an HHNN model as an analog of a phasor CHNN. The proposed HHNN shares common concepts with a CHNN, such as the angle and energy. However, HHNNs and CHNNs are different in many aspects. The states of hyperbolic neurons do not form a circle, and, therefore, the start and end states are not identical. In the quantized version, unlike complex-valued neurons, hyperbolic neurons can have an infinite number of states. The rest of this brief is organized as follows. Section II provides an introduction to CHNNs, which is used as an outline for constructing HHNNs. Section III describes hyperbolic algebra, which is necessary for constructing an HHNN. Then, we construct an HHNN in Section IV. Finally, Section V concludes this brief. II. CHNN S In this section, we briefly describe the conventional theory behind CHNNs [16]–[18]. The aim of this brief is to construct HHNNs as an analog of CHNNs. This section provides an outline for constructing HHNNs given in Section IV.
2162–237X/$31.00 © 2012 IEEE
336
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
imaginary
follows: E =−
input output
−1
1
It is easy to prove E = E. Therefore, E is a real number. When a neuron is updated, the energy always decreases in the wider sense. For state vector z = (z 1 , z 2 , . . . , z N ), consider the rotated state eiθ z = (eiθ z 1 , eiθ z 2 , . . . , eiθ z N ). Then, the energy of the rotated state is as follows: 1 iθ E =− e z k wkj eiθ z j (5) 2
real
−i
Fig. 1. Activation function of complex-valued neurons. The output is the intersection of the complex unit circle and the line passing through the origin and input.
(1)
If z is expressed in the polar form, r exp(i θ ), then f (z) = exp(i θ ). Fig. 1 illustrates the activation function f (z). From a geometric point of view, f (z) is the complex number on the complex unit circle closest to z. Let z r and z be the real part and complex conjugate of z, respectively. Then, f (z) is the complex number c such that c maximizes (cz)r , where c runs over the complex unit circle. The set of complex-valued states of a neuron is cyclic. Suppose that the target data range is [0, 1]. We typically map the target values x to the state exp(i (2π x)) of complex-valued neurons. Then, both 0 and 1 are mapped to the same state. However, the two ends should be mapped to different states. This is a problem caused by complex-valued neurons. B. CHNNs Each neuron of a CHNN is connected to all other neurons. Let the connection weight from neuron j to neuron k be denoted by wkj . Then, the following relation is required to ensure that a CHNN reaches a stable state: wkj = w j k .
(2)
We denote the state of neuron j as z j . Then, the weighted sum input Ik to neuron k is expressed as follows: Ik = wkj z j . (3) j =k
C. Energy Function Let the state of a CHNN be z = (z 1 , z 2 , . . . , z N ), where N is the number of neurons. Then, the energy E is defined as
j =k
k
j =k
k
j =k
(6)
1 z k wkj z j . 2
(7)
A. Complex-Valued Neurons The input and output of complex-valued neurons are complex numbers. For an input z, the output of a complex-valued neuron is determined by an activation function. In this brief, for a complex number z = x + i y, where i is the imaginary unit, we define an activation function f (z) as follows:
k
1 −iθ =− e z k wkj eiθ z j 2 =−
x + iy z = . |z| x 2 + y2
(4)
j =k
k
i
f (z) =
1 z k wkj z j . 2
Therefore, the energies of a pattern and its rotated patterns are equal. This fact is referred to as rotation invariance [25]. D. Complex-Valued Hebbian Learning Rule The complex-valued Hebbian learning rule is the simplest learning method for a CHNN. Let the pth training pattern p p p vector be denoted by (c1 , c2 , . . . , c N ) ( p = 1, 2, . . . , P), where P is the number of training patterns. Then, the complexvalued Hebbian learning rule produces the following connection weights: p p ck c j . (8) wkj = p
When the qth training pattern is applied to a CHNN, the weighted sum input Ik to neuron k is as follows: p p q Ik = ck c j c j (9) j =k p q
= (N − 1)ck +
p p q
ck c j c j .
(10)
j =k p =q
If the second term of the right-hand side is small enough, we can conclude that the qth training pattern is stable. The second term is referred to as the crosstalk term. E. Quantized CHNNs The quantized version of a CHNN is often applied. The CHNN and the quantized CHNN have different activation functions. Let K be a positive integer greater than two. The activation function f (z) of a quantized complex-valued neuron is defined as follows: ⎧ 1 0 ≤ arg(z) < θ K ⎪ ⎪ ⎪ ⎪ exp(i (2θ )) θ K ≤ arg(z) < 3θ K ⎪ K ⎪ ⎪ ⎪ exp(i (4θ )) 3θ ⎪ K K ≤ arg(z) < 5θ K ⎪ ⎪ ⎨ .. .. . (11) f (z) = . ⎪ )) (2K − 3)θ K ≤ arg(z) exp(i (2(K −1)θ ⎪ K ⎪ ⎪ ⎪ < (2K − 1)θ K ⎪ ⎪ ⎪ ⎪ 1 (2K − 1)θ K ≤ arg(z) ⎪ ⎪ ⎩ < 2π
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
337
unipotent
imaginary input
u
i output −1
−1
real
1
1
real
−u
−i
Fig. 2. Complex-valued activation function (K = 8). Dotted lines represent the decision boundaries.
where θ K = π/K . Hence, the number of complex-valued states of a neuron is K . Fig. 2 illustrates the activation function of quantized complex-valued neurons in the case of K = 8. The dotted lines represent the decision boundaries. Because quantized complex-valued neurons have multiple states, quantized CHNNs are often applied to multilevel data such as gray-scale images. Before using a CHNN, we have to select and fix the value of K . If a new data element exceeds K , we cannot apply the data to the CHNN. Thus, quantized CHNNs lack flexibility.
Fig. 3. Domain D = {x + uy; x > 0, x 2 > y 2 } and curve S = {x + uy ∈ D; x 2 − y 2 = 1}.
follows: D = {z = x + uy ∈ H; x > 0, x 2 > y 2 }.
(16)
Domain D is the right hyperbolic quadrant of the hyperbolic plane. The following subset S of D is of particular importance: S = {z ∈ D; |z| = 1}.
(17)
Fig. 3 illustrates domain D and the curve representing S on the hyperbolic plane. For a hyperbolic number ut, where t is a real number, we define the exponential function exp ut as follows:
III. H YPERBOLIC N UMBERS
exp ut = cosh t + u sinh t.
In this section, we describe hyperbolic numbers. A unipotent number u is one that simultaneously satisfies the algebraic properties u = ±1 and u 2 = 1. Hyperbolic numbers are written in the form x + uy, where x and y are real numbers. The set of all hyperbolic numbers is denoted by H. Hyperbolic numbers are 2-D vectors over the field of real numbers, and are identified by points in the 2-D real plane. This 2-D real plane is referred to as the hyperbolic plane. Addition and multiplication in H are defined as follows:
Proposition 3.1: S = {exp ut; t ∈ R}. Proof: By corresponding z = x + uy ∈ S with t = log(x + y), we complete the proof. Any element z = x + uy ∈ D is uniquely represented in the hyperbolic polar form as
(x + uy) + (x + y u) = (x + x ) + u(y + y )
(12)
(x + uy)(x + uy ) = (x x + yy ) + u(x y + x y). (13) Multiplication is clearly commutative. For a hyperbolic number z = x + uy, the real numbers x and y are referred to as the real and unipotent parts of z, respectively. In this brief, the real part of z is denoted by z r . The hyperbolic conjugate z and the hyperbolic modulus |z| of a hyperbolic number z = x + uy are defined as follows: z = x − uy ! |z| = |zz| = |x 2 − y 2 |.
(14) (15)
It is easy to prove that z 1 z 2 = z 1 z 2 for any hyperbolic numbers z 1 and z 2 . In this brief, we do not need to consider the entire range of hyperbolic numbers. We require only the domain D defined as
(18)
z = r exp(ut) (19) ! (20) r = |z| = x 2 − y 2 y t = tanh−1 . (21) x Parameter t is referred to as the hyperbolic angle. Proposition 3.2: For hyperbolic numbers z 1 = r1 exp ut1 and z 2 = r2 exp ut2 ∈ D, the following equation holds: z 1 z 2 = r1r2 exp u(t1 + t2 ).
(22)
Proof: z 1 z 2 = r1r2 exp ut1 exp ut2
(23)
= r1r2 (cosh t1 + u sinh t1 )(cosh t2 + u sinh t2 ) (24) = r1r2 ((cosh t1 cosh t2 + sinh t1 sinh t2 ) +u(cosh t1 sinh t2 + sinh t1 cosh t2 )) = r1r2 (cosh(t1 + t2 ) + u sinh(t1 + t2 ))
(25) (26)
= r1r2 (exp u(t1 + t2 )).
(27)
338
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
unipotent
unipotent
u input output
real
−1
1
real
−u
Fig. 4. Hyperbolic rotation. Through multiplication by exp ut, a point in D moves while following the hyperbolic modulus.
Fig. 5. Hyperbolic activation function. The output is the intersection of curve S and the line passing through the origin and the input.
IV. HHNN S Multiplying by exp ut follows the hyperbolic modulus and is referred to as hyperbolic rotation (Fig. 4). Proposition 3.3: D is closed under addition. Proof: Trivial. Proposition 3.4: D is a multiplicative group. Proof: From Proposition 3.2, D is closed under multiplication. Hyperbolic algebra includes the associative law. It is clear that unity (1) belongs to D. The inverse of z = r exp ut ∈ D is r −1 exp(−ut). Therefore, D is a multiplicative group. Proposition 3.5: For any z ∈ S, z belongs to S, and is the inverse of z. Proof: For z = exp ut, we show that z = exp u(−t). z = cosh t − u sinh t = cosh(−t) + u sinh(−t) = exp u(−t).
(28) (29) (30)
Hence, we proved that z ∈ S. From Proposition 3.2, we get the following equation: zz = exp ut exp u(−t) = exp u(t − t) = exp u0 = 1.
A. Hyperbolic Neuron In this subsection, we define hyperbolic neurons. Inputs of hyperbolic neurons are hyperbolic numbers in D. Outputs of hyperbolic neurons are hyperbolic numbers in S. For an input z, a hyperbolic neuron with activation function g(z) produces the output g(z). For z = x + uy ∈ D, we define the activation function g(z) as follows: g(z) =
x + uy z = . |z| x 2 − y2
(34)
We call this activation function g(z) the hyperbolic activation function. When we represent z = r exp ut in hyperbolic polar form, we can write g(z) = exp ut. From a geometrical point of view, g(z) is the intersection of curve S and the line passing through the origin and point z on the hyperbolic plane. Fig. 5 illustrates the relation between z and g(z). The following proposition is of particular importance. Proposition 4.1: For z ∈ D, g(z) is equal to c ∈ S and c minimizes (cz)r . Proof: We write z and c in hyperbolic polar form as follows:
(31)
z = r exp utz
(35)
(32) (33)
c = exp utc .
(36)
Therefore, z is the inverse of z. Proposition 3.6: S is a multiplicative group. Proof: We prove that S is closed under multiplication. For any z 1 = exp ut1 and z 2 = exp ut2 ∈ S, equation z 1 z 2 = exp u(t1 + t2 ) holds. Thus we proved that z 1 z 2 ∈ S. Hyperbolic algebra includes the associative law. It is clear that unity (1) belongs to D. From Proposition 3.5, the inverse of z exists for any z ∈ S. This concludes the proof. We can find some correspondences between the complex plane and domain D in the hyperbolic plane. Domain S, which is the set of points with modulus 1, corresponds to the complex unit circle. The hyperbolic polar form and hyperbolic angle correspond to the polar form and phase of a complex plane, respectively.
Then, we get g(z) = exp utz . Moreover (cz)r = (r exp u(tz − tc ))r = r cosh(tz − tc ).
(37) (38)
If and only if tz = tc , (cz)r is minimized. Therefore, c = exp utz minimizes (cz)r , which concludes the proof. Suppose that the target data belongs in the range [0, 1]. In the case of a complex-valued neuron, both 0 and 1 are typically mapped to the same state. In the case of a hyperbolic neuron, if we map the target value x to exp u(2x − 1)A, where A is an arbitrary positive number, 0 and 1 are mapped to the most different states exp u(−A) and exp u A, respectively. Then, the area used by the states of the neuron spans from exp u(−A) to exp u A. Thus, for large values of A we can use a wider range.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
B. HHNNs In this subsection, we construct a HHNN. First, we define connection weights. Every neuron of the HHNN is connected to all other neurons. We denote the connection weight from neuron j to neuron k by wkj . The connection weights must satisfy the following conditions: 1) wkj ∈ D; 2) wkj = w j k . We denote the output of neuron k by z k . We define the weighted sum input Ik to neuron k as follows: wkj z j . (39) Ik = j =k
Thus, Ik belongs to D. Neuron k receives the weighted sum input Ik and produces g(Ik ).
Let the state of HHNN be z = (z 1 , z 2 , . . . , z N ), where N is the number of neurons. We define the energy function E of HHNN as follows: 1 E = z k wkj z j . (40) 2 k
j =k
It is easy to prove E = E. Therefore, E is a real number. Suppose that the state z l of neuron l changes to z l . Then, the energy gap E is as follows: ⎞ ⎛ 1 ⎝ z k wkl z l + z l wl j z j ⎠ E = 2 k =l j =l ⎞ ⎛ 1 − ⎝ z k wkl z l + z l wl j z j ⎠ (41) 2 k =l j =l ⎞r ⎛ ⎞r ⎛ z l wl j z j ⎠ − ⎝ z l wl j z j ⎠ (42) =⎝ j =l
= ⎝z l
⎞r
⎛
j =l
wl j z j ⎠ − ⎝z l
j =l
⎞r wl j z j ⎠
(43)
j =l
= (z l Il )r − (z l Il )r .
(44)
From Proposition 4.1, z l minimizes (z Il )r . Hence, E is nonpositive, and E does not increase. Because j =k z k wkj z j belongs to D, the energy is definitely positive. Therefore, the energy always converges. For state vector z = (z 1 , z 2 , . . . , z N ), consider the hyperbolic rotated state eut z = (eut z 1 , eut z 2 , . . . , eut z N ). Then, the energy of the hyperbolic rotated state is as follows: 1 ut e z k wkj eut z j (45) E = 2 k
j =k
k
j =k
k
j =k
1 u(−t ) = e z k wkj eut z j 2 =
Therefore, the energies of a pattern and its hyperbolic rotated patterns are equal. This property is called hyperbolic rotation invariance. D. Hyperbolic Hebbian Learning Rule Denote the state of neuron j of the pth training pattern by p c j . Then, we define the connection weights as follows: p p ck c j . (48) wkj = p
We call this learning rule the hyperbolic Hebbian learning rule. p From Propositions 3.3 and 3.4, and the fact that c j ∈ D, wkj ∈ D is true. When the qth training pattern is given, the weighted sum input Ik to neuron k is as follows: p p q ck c j c j (49) Ik = j =k p
C. Energy Function
⎛
339
1 z k wkj z j . 2
(46) (47)
q
= (N − 1)ck +
p p q
ck c j c j .
(50)
j =k p =q
If the crosstalk terms are small enough, we can conclude that the qth training pattern is stable as in the case of CHNNs. p p q However, the crosstalk term is large. From ck c j c j ∈ S, the real part of the crosstalk term is greater than (N − 1)(P − 1). Therefore, hyperbolic Hebbian learning rule cannot deal with multiple patterns. E. Quantized HHNNs In this section, we consider quantizing the HHNN. We take a positive number T and call it a hyperbolic resolution factor. We take an infinite number of discrete points exp u(nT ) on S, where n is an integer, and define the set {exp u(nT )}∞ n=−∞ to be the states of the quantized hyperbolic neurons. Next, we have to create a quantized version of the activation function to decrease the energy function. For a hyperbolic number z to decrease the energy, it suffices to set the activation function to exp u(nT ), where n minimizes (z exp u(−nT ))r , because then expression (44) is nonpositive. Next, we determine the decision boundary between the states exp u(nT ) and exp u(n + 1)T . A point z on the decision boundary satisfies (z exp u(−nT ))r = (z exp u(−n − 1)T )r . By expressing z = r exp ut, we obtain the following equations: (r exp u(t − nT ))r = (r exp u(t − (n + 1)T ))r
(51)
cosh(t − nT ) = cosh(t − (n + 1)T ) (52) t − nT = (n + 1)T − t (53) 1 (54) t = (n + )T. 2 Thus, we obtain the decision boundary z = a exp u(n +1/2)T , where a is a positive parameter. For z = r exp ut ∈ D, if (n − 1/2)T < t < (n + 1/2)T , then f (z) = exp u(nT ). In the case of complex-valued neurons, we have to fix the number of states K . If a new data element exceeds the fixed range, we cannot apply the data to the CHNN. Because hyperbolic neurons have an infinite number of states, the range of data is not restricted.
340
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
unipotent
geometric properties. In the future, we will discuss our model from the geometric point of view to find applicable targets, such as text categorization and semantic navigation, which need information processing in non-Euclidean space.
u input output −1
1
R EFERENCES real
−u
Fig. 6. Quantized hyperbolic activation function. Dotted lines are the decision boundaries.
If we want to restrict the number of states to be finite, we can take a finite number of sequential integers n. Fig. 6 illustrates the states of a quantized hyperbolic neuron in the case of five states. The range of n is from −2 to 2. The dotted lines are the decision boundaries. V. C ONCLUSION In this brief, we constructed an HHNN as an analog of a CHNN. HHNNs share many common concepts with CHNNs such as the angle, rotation invariance, energy function, and so on. On the other hand, HHNNs posses some properties that are better than those of CHNNs. 1) The set of complex-valued states of a neuron forms a circle, and, thus, it is hard to properly map the target data to CHNN; on the other hand, the set of hyperbolic states of a neuron is homeomorphic to the real number line, and, thus, it is easy to naturally map the target data to a HHNN. 2) Unlike quantized complex-valued neurons, quantized hyperbolic neurons have an infinite number of states; CHNNs can handle only a fixed range of data, while HHNNs do not restrict the range of data. If the range of the target data is difficult to predict beforehand, HHNN may be suitable. In our construction, hyperbolic neurons receive only hyperbolic numbers belonging to the right quadrant of the hyperbolic plane. Since hyperbolic Hebbian learning rule does not work well, the performance of HHNNs is worse than that of CHNNs at present, though they have some better properties. So we have to develop practical learning methods. Projection rule and gradient descent learning rule for CHNN have been proposed [19], [26]– [28]. We should develop those for HHNN. Then, it is possible that connection weights do not belong to the right quadrant. In this case, the hyperbolic activation function should be extended to the entire hyperbolic plane. HHNNs are expected to be applicable when constructing higher dimensional Clifford HNNs. For example, Isokawa et al. [24] attempted to construct an HNN with commutative quaternions. Their proposed model requires the use of a HHNN, but their work is still incomplete. In this brief, we mainly discussed our model from the algebraic point of view. Hyperbolic algebra has excellent
[1] S. Buchholz, “A theory of neural computation with Clifford algebra,” Ph.D. dissertation, Dept. Comput. Sci., Univ. Kiel, Kiel, Germany, 2005. [2] N. Benvenuto and F. Piazza, “Multi-valued and universal binary neurons: Theory, learning and application,” IEEE Trans. Signal Process., vol. 40, no. 4, pp. 967–969, May 1992. [3] T. Nitta, “Complex numbered version of the back-propagation algorithm,” in Proc. INNS World Congr. Neural Netw., vol. 3, pp. 576–579, 1993. [4] T. Nitta, “An extension of the back-propagation algorithm to complex numbers,” Neural Netw., vol. 10, no. 8, pp. 1391–1415, 1997. [5] A. Hirose, “Continuous complex-valued back-propagation learning,” Electron. Lett., vol. 28, no. 20, pp. 1854–1855, 1992. [6] T. Nitta, “An extension of the back-propagation algorithm to quaternions,” in Proc. Int. Conf. Neural Inf. Process., vol. 1. 1996, pp. 247–250. [7] M. Kobayashi, J. Muramatsu, and H. Yamazaki, “Construction of high-dimensional neural networks by linear connections of matrices,” Electron. Commun. Japan III, vol. 86, no. 3, pp. 38–45, 2003. [8] P. Arena, L. Fortuna, G. Muscato, and M. G. Xibilia, Neural Networks in Multidirectional Domains. London, U.K.: Springer, 1998. [9] H. Ritter, “Self-organizing maps on non-euclidean spaces,” Kohonen Maps, E. Oja, S. Kaski, Eds. Amsterdam, The Netherlands: Elsevier, pp. 97–110, 1999. [10] J. Ontrup and H. Ritter, “Hyperbolic self-organizing maps for semantic navigation,” Advances in Neural Information Processing Systems, vol. 14. Cambridge, MA: MIT Press, 2001, pp. 1417–1424. [11] J. Ontrup and H. Ritter, “Text categorization and semantic browsing with self-organizing maps on non-euclidean spaces,” in Proc. 5th Eur. Conf. Principles Pract. Knowl. Discovery Databases, 2001, pp. 338–349. [12] S. Buchholz and G. Sommer, “A hyperbolic multilayer perceptron,” in Proc. IEEE-INNS-ENNS Int. Joint Conf. Neural Netw., vol. 2. Feb. 2000, pp. 129–133. [13] T. Nitta and S. Buchholz, “On the decision boundaries of hyperbolic neurons,” in Proc. Int. Joint Conf. Neural Netw., 2008, pp. 2973–2979. [14] J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities,” Proc. Nat. Acad. Sci. United States Amer., vol. 79, no. 8, pp. 2554–2558, 1982. [15] S. Jankowski, A. Lozowski, and J. M. Zurada, “Complex-valued multistate neural associative memory,” IEEE Trans. Neural Netw., vol. 7, no. 6, pp. 1491–1496, Nov. 1996. [16] A. J. Noest, “Phasor neural networks,” in Neural Information Processing Systems, D. Z. Anderson, Ed. New York: AIP, 1988, pp. 584–591. [17] A. J. Noest, “Discrete-state phasor neural networks,” Phys. Rev. A, vol. 38, no. 4, pp. 2196–2199, 1988. [18] I. N. Aizenberg, N. N. Aizenberg, and J. Vandewalle, Multi-Valued and Universal Binary Neurons: Theory, Learning and Application. Boston, MA: Kluwer Academic, 2000. [19] H. Aoki, “A complex-valued neuron to transform gray level images to phase information,” in Proc. Int. Conf. Neural Inf. Process., vol. 3, pp. 1084–1088, 2002. [20] M. K. Muezzinoglu, C. Guzelis, and J. M. Zurada, “A new design method for the complex-valued multistate Hopfield associative memory,” IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 891–899, Jul. 2003. [21] T. Isokawa, H. Nishimura, N. Kamiura, and N. Matsui, “Associative memory in quaternionic Hopfield neural network,” Int. J. Neural Syst., vol. 18, no. 2, pp. 135–145, 2008. [22] T. Isokawa, H. Nishimura, and N. Matsui, “An iterative learning scheme for multistate complex-valued and quaternionic Hopfield neural networks,” in Proc. Int. Joint Conf. Neural Netw., pp. 1365–1371, 2009. [23] Y. Kuroe, S. Tanigawa, and H. Iima, “Models of Hopfield-type Clifford neural networks and their energy functions - hyperbolic and dual valued networks,” in ICONIP Part I (Lecture Notes in Computer Science), B. L. Lu, L. Zhang, and J. Kwok, Eds. Berlin, Germany: SpringerVerlag, 2011, pp. 560–569.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 24, NO. 2, FEBRUARY 2013
[24] T. Isokawa, H. Nishimura, and N. Matsui, “Commutative quaternion and multistate hopfield neural networks,” in Proc. Int. Joint Conf. Neural Netw., pp. 1281–1286, 2010. [25] R. S. Zemel, C. K. I. Williams, and M. C. Mozer, “Lending direction to neural networks,” Neural Netw., vol. 8, no. 4, pp. 503–512, 1995. [26] M. Kitahara and M. Kobayashi, “Projection rules for complex-valued associative memory with large constant terms,” Nonlinear Theory Appl., vol. 3, no. 3, pp. 426–435, 2012.
341
[27] M. Kobayashi, H. Yamada, and M. Kitahara, “Noise robust gradient descent learning for complex-valued associative memory,” IEICE Trans. Fundamentals Electron., Commun. Comput. Sci., vol. E94-A, no. 8, pp. 1756–1759, 2011. [28] D. L. Lee, “Improving the capacity of complex-valued neural networks with a modified gradient descent learning rule,” IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 439–443, Mar. 2001.
