Attractor comparisons based on density T. L. Carroll Citation: Chaos: An Interdisciplinary Journal of Nonlinear Science 25, 013111 (2015); doi: 10.1063/1.4906342 View online: http://dx.doi.org/10.1063/1.4906342 View Table of Contents: http://scitation.aip.org/content/aip/journal/chaos/25/1?ver=pdfcov Published by the AIP Publishing Articles you may be interested in Generative model selection using a scalable and size-independent complex network classifier Chaos 23, 043127 (2013); 10.1063/1.4840235 Combining multiple models to generate consensus: Application to radiation-induced pneumonitis prediction Med. Phys. 35, 5098 (2008); 10.1118/1.2996012 Genetic Algorithm Based Incremental Learning For Optimal Weight and Classifier Selection AIP Conf. Proc. 952, 258 (2007); 10.1063/1.2816630 Bayesian Evidence Framework for Decision Tree Learning AIP Conf. Proc. 803, 88 (2005); 10.1063/1.2149783 Experimenting with rule induction algorithms in HEP data analysis AIP Conf. Proc. 583, 107 (2001); 10.1063/1.1405276
This article is copyrighted as indicated in the article. Reuse of AIP content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP: 202.28.191.34 On: Sun, 08 Mar 2015 18:46:13
CHAOS 25, 013111 (2015)
Attractor comparisons based on density T. L. Carrolla) US Naval Research Lab, Washington, DC 20375, USA
(Received 10 September 2014; accepted 9 January 2015; published online 16 January 2015) Recognizing a chaotic attractor can be seen as a problem in pattern recognition. Some feature vector must be extracted from the attractor and used to compare to other attractors. The field of machine learning has many methods for extracting feature vectors, including clustering methods, decision trees, support vector machines, and many others. In this work, feature vectors are created by representing the attractor as a density in phase space and creating polynomials based on this density. Density is useful in itself because it is a one dimensional function of phase space position, but representing an attractor as a density is also a way to reduce the size of a large data set before analyzing it with graph theory methods, which can be computationally intensive. The density computation in this paper is also fast to execute. In this paper, as a demonstration of the usefulness of density, the density is used directly to construct phase space polynomials for comparing attractors. Comparisons between attractors could be useful for tracking changes in an experiment when the underlying equations are too complicated for vector field modeling. [http://dx.doi.org/10.1063/1.4906342] Attractors from chaotic and some non-chaotic dynamical systems form patterns in phase space that reflect the dynamics. While much work has been devoted to extracting the dynamics from these patterns, the concern in this paper is simply to find a number that indicates in some way how different the attractors are, and some way to predict how this difference statistic will change as some parameter in the dynamical system changes. One could describe this problem as pattern recognition. Most pattern recognition algorithms proceed by creating a set of feature vectors for a set of data, and then comparing feature vectors. In this work, feature vectors are created by representing the attractor as a density in phase space. A set of one dimensional polynomials are created from this density using Gram-Schmidt orthogonalization. As a one dimensional function of multiple variables, the density function can be thought of as a manifold. The phase space density as a function of position reflects the local dynamics, so calculating the density at different spatial resolutions can be used to compare attractors at different length scales. Feature vectors for an attractor are created by projecting the density for that attractor onto the phase space polynomials. Many dynamical systems that output low dimensional signals have very complicated descriptions, making vector field modeling impractical, so comparisons based on the shape of the attractor can be useful.
I. INTRODUCTION
The first methods for characterizing chaotic attractors focussed on studying their geometry.1–6 Measurement of dimension was one of the first topics of interest, but other geometrical methods such as templates were also considered. Some study of geometry continues, but most analysis of a)
Electronic mail: [email protected]
1054-1500/2015/25(1)/013111/6/$30.00
chaotic attractors has shifted to prediction of time series and fitting of vector fields. Finding the vector field for a chaotic system gives the simplest possible description of the system, but finding a simple vector field is difficult. If the functional form of the vector field is known, various methods can be used to find the parameters for the particular system under study;7–10 for more general situations, fitting a vector field may require many parameters and a complicated fitting process. Fitting of vector fields is also sensitive to noise. In some papers, time series prediction is used to find vector fields for chaotic systems,11–13 but sensitive dependence on initial conditions makes long term prediction impossible. More recently, chaotic signals have been represented as networks,14 making it possible to characterize a chaotic system using the tools of network analysis. Direct comparison of chaotic attractors is possible,15 but that paper could not say, how the comparison would change if one of the attractors changed. The shape of a chaotic attractor is well defined and not subject to prediction errors, but there are few tools for describing this shape.4,16 One way to describe this shape is by describing the attractor as a probability distribution in phase space. It is well known that the probability measure of a dynamical system reflects it’s long term behavior in phase space.5,17 While considering the attractor as a distribution means giving up on prediction, prediction is not always the goal of analyzing an attractor. If I am trying to find how similar two systems are to each other, then knowing the density makes it possible to compare attractors using only a small number of parameters. For some dynamical systems, such as complex electronic circuits, or driven structures, it is not mathematically tractable to generate a model for the system,18,19 but if all I want to do is to compare the output from different systems, knowing the exact equations is not necessary. Representing an attractor as a density reduces the size of the data set, which can be important for graph theory
25, 013111-1
This article is copyrighted as indicated in the article. Reuse of AIP content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP: 202.28.191.34 On: Sun, 08 Mar 2015 18:46:13
013111-2
T. L. Carroll
methods of analysis.14,20 Graph theory methods often represent a data set as an adjacency matrix, which requires comparing each point in the data set to every other point in the data set. Reducing the size of the data set may be necessary to make graph theory computations practical, and the density computation method described here is fast to execute, making it useful for large data sets. It is common in dynamics to sweep some parameter of a dynamical system to map out a bifurcation diagram in order to understand how the dynamical system changes as the parameter changes. In this work, it is assumed that the changing parameter does not take the system through any bifurcations, or the observer has limited or no control over the system under observation. It may be that all that is available to the observer is a set of output signals, so the goal is to embed these signals in a phase space and assign a number that describes how different the resulting attractors are from each other. A. Range of application
In this work, the aim is to show that the attractor density by itself is useful for comparing attractors. Combining density with graph theory methods will be left for later work. Representing an attractor as a density requires dividing the phase space into bins and calculating a histogram for the attractor. Binning the phase space limits the resolution with which the attractor is represented, but this same limit is present for other measurements on chaotic data. Dimensions or Lyapunov exponents, for example, are calculated by comparing a set of near neighbor points in the phase space, which means that the attractor properties are being calculated for a ball or box in the space. Calculations based on phase space neighbors also limit the resolution with which the attractor is represented. Nearest neighbor searches are also slower than computing a histogram. The histogram used to calculate the phase space density has some similarity to box counting methods,5 which have been used to calculate dimension. The number of filled bins could be used for box counting, but for the density, the number of points in each bin and the spatial location of each bin are the important quantities- these numbers are not used in box counting. While attractor density could be used to differentiate between very different attractors such as a Lorenz and a Rossler, the density method is best suited for tracking small changes in an attractor as a way to detect parameter changes in a chaotic system. In one example below, density comparisons are used to track a changing parameter in a Rossler circuit. In a second example, density comparisons are used to detect nonlinearity in a chaotically driven op amp circuit. Characterizing nonlinearity in electronic circuits by driving them with a test signal is an important application.18 Similar techniques have been used to characterize communications channels or driven structures.19 Because the density method reduces the size of the data set in a computationally fast manner, it is useful for this sort of characterization when the data sets are large.
Chaos 25, 013111 (2015)
II. FINDING DENSITIES
In order to find the phase space density for a time series s, the time series is first embedded in a d dimensional phase space using the method of delays.21 For each point in s, a vector sðiÞ is defined as sðiÞ ¼ fsðiÞ; sði þ sÞ; …sði þ ðd 1ÞsÞg. The embedding dimension d and the delay s may be found by any one of a number of standard methods.21 Each of the d dimensional points is sorted into a bin by assigning a bin number. If the maximum and minimum values of s are smax and smin, and there are Nb bins along each dimension of the phase space, then the bin number kb far a point s(i) is kb ¼
d X ½ðsðiÞ smin Þ=ðsmax smin ÞNbj1 :
(1)
j¼1
For a large number of bins Nb, trying to store the number of points in every bin would require a very large amount of memory. It is not necessary to keep track of all the bins, because most of the bins are empty, so instead, a list is kept of bin numbers that contain points. The list of filled bins is kept in L, while the number of points in each filled bin is kept in P. Every time a bin number kb is found, the list L is scanned to see if the particular kb has been found before. If it has been found before at the location i in the list, then P(i) is incremented by 1; if the particular value of kb has not been found before, then kb is added to the end of the list L. Once all N points in s have been assigned to bins, the density qðiÞ is calculated as qðiÞ ¼ PðiÞ=N. Creating the phase space histogram is similar to box counting, but the histogram goes beyond box counting. In box counting, only the number of filled boxes is used; in this histogram, it is not the number of boxes that is counted but the number of points in each box and it’s location. The range of bin numbers kb is limited by the number of bits used to represent integers on the computer, so if the number of bins or the embedding dimension is too large, it may be necessary to represent kb as a multidimensional number. III. CREATING FEATURE VECTORS
There are many tools for creating feature vectors from 1-dimensional signals, but finding features for complicated multi-dimensional probability distributions is more difficult. Even in multiple dimensions, numbers such as mean, standard deviation, various statistical moments, etc., do not provide an accurate representation of an attractor. One way to create feature vectors is to project the attractor density onto a set of basis functions. The most efficient basis functions are functions that occupy roughly the same part of phase space as the attractor being characterized. In this paper, one attractor is chosen as an index attractor to which other attractors are compared. A set of orthogonal polynomials are created from the density for the index attractor, and the densities for other attractors may be projected onto these polynomials. Giona et al.10 has demonstrated the construction of orthogonal polynomials from a chaotic time
This article is copyrighted as indicated in the article. Reuse of AIP content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP: 202.28.191.34 On: Sun, 08 Mar 2015 18:46:13
013111-3
T. L. Carroll
Chaos 25, 013111 (2015)
series in order to model the vector field; in this paper, a similar procedure is applied to the density. Given the phase space density q created from a density with Nf filled bins, the density polynomials pj, j ¼ 0:::Nord are given by Pj ðiÞ ¼ qðiÞj
j1 X
hpm ; qj ipm ðiÞ;
m¼0
(2)
pj ðiÞ ¼ Pj ðiÞ=kPj ðiÞk; where kk represents the Euclidean norm, and the inner product is given by hpm ; qi ¼
Nf X
pm ðiÞqðiÞ:
(3)
i¼1
A. Comparing attractors
Creating the polynomials pj is the training phase for the pattern recognition problem. Different attractors may now be compared to these polynomials to generate feature vectors. A signal su from an unknown attractor may be embedded in phase space with the same embedding parameters as the signal s, and the phase space density qu calculated in the same manner as in Eq. (1) and the subsequent paragraph. The feature vector for su will be C ¼ ½c0 ; c1 ; :::cNord , where the cj’s are the projection coefficients found by projecting qu onto the polynomials pj. The list of densities qu is indexed by its position in the list Lu of filled bins. For each density value qu ðiÞ, there is a corresponding bin number kb ðuÞ. The list L must be searched to see if it contains the same bin number. If LðlÞ ¼ kb ðuÞ, then the variable cj ðiÞ is set to pðlÞ; if kb ðuÞ is not in the list L, then cj ðiÞ ¼ 0. The projection coefficients are found as cj ¼
N f ðuÞ X
qu ðiÞcj ðiÞ;
(4)
i¼1
where Nf ðuÞ is the number of filled bins for the unknown attractor. IV. ROSSLER CIRCUIT EXPERIMENT
As an illustration of the density method, the projection coefficient vectors are used to detect parameter variations in a Rossler circuit. The Rossler circuit here was originally described in Ref. 22. The circuit was described by the equations dx ¼ 104 ½ 0:05x 0:5y z; dt dy ¼ 104 ½ ax þ 0:128y; dt dz ¼ 104 ½z þ gð xÞ; dt ( ) 0 x
This article is copyrighted as indicated in the article. Reuse of AIP content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP: 202.28.191.34 On: Sun, 08 Mar 2015 18:46:13
CHAOS 25, 013111 (2015)
Attractor comparisons based on density T. L. Carrolla) US Naval Research Lab, Washington, DC 20375, USA
(Received 10 September 2014; accepted 9 January 2015; published online 16 January 2015) Recognizing a chaotic attractor can be seen as a problem in pattern recognition. Some feature vector must be extracted from the attractor and used to compare to other attractors. The field of machine learning has many methods for extracting feature vectors, including clustering methods, decision trees, support vector machines, and many others. In this work, feature vectors are created by representing the attractor as a density in phase space and creating polynomials based on this density. Density is useful in itself because it is a one dimensional function of phase space position, but representing an attractor as a density is also a way to reduce the size of a large data set before analyzing it with graph theory methods, which can be computationally intensive. The density computation in this paper is also fast to execute. In this paper, as a demonstration of the usefulness of density, the density is used directly to construct phase space polynomials for comparing attractors. Comparisons between attractors could be useful for tracking changes in an experiment when the underlying equations are too complicated for vector field modeling. [http://dx.doi.org/10.1063/1.4906342] Attractors from chaotic and some non-chaotic dynamical systems form patterns in phase space that reflect the dynamics. While much work has been devoted to extracting the dynamics from these patterns, the concern in this paper is simply to find a number that indicates in some way how different the attractors are, and some way to predict how this difference statistic will change as some parameter in the dynamical system changes. One could describe this problem as pattern recognition. Most pattern recognition algorithms proceed by creating a set of feature vectors for a set of data, and then comparing feature vectors. In this work, feature vectors are created by representing the attractor as a density in phase space. A set of one dimensional polynomials are created from this density using Gram-Schmidt orthogonalization. As a one dimensional function of multiple variables, the density function can be thought of as a manifold. The phase space density as a function of position reflects the local dynamics, so calculating the density at different spatial resolutions can be used to compare attractors at different length scales. Feature vectors for an attractor are created by projecting the density for that attractor onto the phase space polynomials. Many dynamical systems that output low dimensional signals have very complicated descriptions, making vector field modeling impractical, so comparisons based on the shape of the attractor can be useful.
I. INTRODUCTION
The first methods for characterizing chaotic attractors focussed on studying their geometry.1–6 Measurement of dimension was one of the first topics of interest, but other geometrical methods such as templates were also considered. Some study of geometry continues, but most analysis of a)
Electronic mail: [email protected]
1054-1500/2015/25(1)/013111/6/$30.00
chaotic attractors has shifted to prediction of time series and fitting of vector fields. Finding the vector field for a chaotic system gives the simplest possible description of the system, but finding a simple vector field is difficult. If the functional form of the vector field is known, various methods can be used to find the parameters for the particular system under study;7–10 for more general situations, fitting a vector field may require many parameters and a complicated fitting process. Fitting of vector fields is also sensitive to noise. In some papers, time series prediction is used to find vector fields for chaotic systems,11–13 but sensitive dependence on initial conditions makes long term prediction impossible. More recently, chaotic signals have been represented as networks,14 making it possible to characterize a chaotic system using the tools of network analysis. Direct comparison of chaotic attractors is possible,15 but that paper could not say, how the comparison would change if one of the attractors changed. The shape of a chaotic attractor is well defined and not subject to prediction errors, but there are few tools for describing this shape.4,16 One way to describe this shape is by describing the attractor as a probability distribution in phase space. It is well known that the probability measure of a dynamical system reflects it’s long term behavior in phase space.5,17 While considering the attractor as a distribution means giving up on prediction, prediction is not always the goal of analyzing an attractor. If I am trying to find how similar two systems are to each other, then knowing the density makes it possible to compare attractors using only a small number of parameters. For some dynamical systems, such as complex electronic circuits, or driven structures, it is not mathematically tractable to generate a model for the system,18,19 but if all I want to do is to compare the output from different systems, knowing the exact equations is not necessary. Representing an attractor as a density reduces the size of the data set, which can be important for graph theory
25, 013111-1
This article is copyrighted as indicated in the article. Reuse of AIP content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP: 202.28.191.34 On: Sun, 08 Mar 2015 18:46:13
013111-2
T. L. Carroll
methods of analysis.14,20 Graph theory methods often represent a data set as an adjacency matrix, which requires comparing each point in the data set to every other point in the data set. Reducing the size of the data set may be necessary to make graph theory computations practical, and the density computation method described here is fast to execute, making it useful for large data sets. It is common in dynamics to sweep some parameter of a dynamical system to map out a bifurcation diagram in order to understand how the dynamical system changes as the parameter changes. In this work, it is assumed that the changing parameter does not take the system through any bifurcations, or the observer has limited or no control over the system under observation. It may be that all that is available to the observer is a set of output signals, so the goal is to embed these signals in a phase space and assign a number that describes how different the resulting attractors are from each other. A. Range of application
In this work, the aim is to show that the attractor density by itself is useful for comparing attractors. Combining density with graph theory methods will be left for later work. Representing an attractor as a density requires dividing the phase space into bins and calculating a histogram for the attractor. Binning the phase space limits the resolution with which the attractor is represented, but this same limit is present for other measurements on chaotic data. Dimensions or Lyapunov exponents, for example, are calculated by comparing a set of near neighbor points in the phase space, which means that the attractor properties are being calculated for a ball or box in the space. Calculations based on phase space neighbors also limit the resolution with which the attractor is represented. Nearest neighbor searches are also slower than computing a histogram. The histogram used to calculate the phase space density has some similarity to box counting methods,5 which have been used to calculate dimension. The number of filled bins could be used for box counting, but for the density, the number of points in each bin and the spatial location of each bin are the important quantities- these numbers are not used in box counting. While attractor density could be used to differentiate between very different attractors such as a Lorenz and a Rossler, the density method is best suited for tracking small changes in an attractor as a way to detect parameter changes in a chaotic system. In one example below, density comparisons are used to track a changing parameter in a Rossler circuit. In a second example, density comparisons are used to detect nonlinearity in a chaotically driven op amp circuit. Characterizing nonlinearity in electronic circuits by driving them with a test signal is an important application.18 Similar techniques have been used to characterize communications channels or driven structures.19 Because the density method reduces the size of the data set in a computationally fast manner, it is useful for this sort of characterization when the data sets are large.
Chaos 25, 013111 (2015)
II. FINDING DENSITIES
In order to find the phase space density for a time series s, the time series is first embedded in a d dimensional phase space using the method of delays.21 For each point in s, a vector sðiÞ is defined as sðiÞ ¼ fsðiÞ; sði þ sÞ; …sði þ ðd 1ÞsÞg. The embedding dimension d and the delay s may be found by any one of a number of standard methods.21 Each of the d dimensional points is sorted into a bin by assigning a bin number. If the maximum and minimum values of s are smax and smin, and there are Nb bins along each dimension of the phase space, then the bin number kb far a point s(i) is kb ¼
d X ½ðsðiÞ smin Þ=ðsmax smin ÞNbj1 :
(1)
j¼1
For a large number of bins Nb, trying to store the number of points in every bin would require a very large amount of memory. It is not necessary to keep track of all the bins, because most of the bins are empty, so instead, a list is kept of bin numbers that contain points. The list of filled bins is kept in L, while the number of points in each filled bin is kept in P. Every time a bin number kb is found, the list L is scanned to see if the particular kb has been found before. If it has been found before at the location i in the list, then P(i) is incremented by 1; if the particular value of kb has not been found before, then kb is added to the end of the list L. Once all N points in s have been assigned to bins, the density qðiÞ is calculated as qðiÞ ¼ PðiÞ=N. Creating the phase space histogram is similar to box counting, but the histogram goes beyond box counting. In box counting, only the number of filled boxes is used; in this histogram, it is not the number of boxes that is counted but the number of points in each box and it’s location. The range of bin numbers kb is limited by the number of bits used to represent integers on the computer, so if the number of bins or the embedding dimension is too large, it may be necessary to represent kb as a multidimensional number. III. CREATING FEATURE VECTORS
There are many tools for creating feature vectors from 1-dimensional signals, but finding features for complicated multi-dimensional probability distributions is more difficult. Even in multiple dimensions, numbers such as mean, standard deviation, various statistical moments, etc., do not provide an accurate representation of an attractor. One way to create feature vectors is to project the attractor density onto a set of basis functions. The most efficient basis functions are functions that occupy roughly the same part of phase space as the attractor being characterized. In this paper, one attractor is chosen as an index attractor to which other attractors are compared. A set of orthogonal polynomials are created from the density for the index attractor, and the densities for other attractors may be projected onto these polynomials. Giona et al.10 has demonstrated the construction of orthogonal polynomials from a chaotic time
This article is copyrighted as indicated in the article. Reuse of AIP content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP: 202.28.191.34 On: Sun, 08 Mar 2015 18:46:13
013111-3
T. L. Carroll
Chaos 25, 013111 (2015)
series in order to model the vector field; in this paper, a similar procedure is applied to the density. Given the phase space density q created from a density with Nf filled bins, the density polynomials pj, j ¼ 0:::Nord are given by Pj ðiÞ ¼ qðiÞj
j1 X
hpm ; qj ipm ðiÞ;
m¼0
(2)
pj ðiÞ ¼ Pj ðiÞ=kPj ðiÞk; where kk represents the Euclidean norm, and the inner product is given by hpm ; qi ¼
Nf X
pm ðiÞqðiÞ:
(3)
i¼1
A. Comparing attractors
Creating the polynomials pj is the training phase for the pattern recognition problem. Different attractors may now be compared to these polynomials to generate feature vectors. A signal su from an unknown attractor may be embedded in phase space with the same embedding parameters as the signal s, and the phase space density qu calculated in the same manner as in Eq. (1) and the subsequent paragraph. The feature vector for su will be C ¼ ½c0 ; c1 ; :::cNord , where the cj’s are the projection coefficients found by projecting qu onto the polynomials pj. The list of densities qu is indexed by its position in the list Lu of filled bins. For each density value qu ðiÞ, there is a corresponding bin number kb ðuÞ. The list L must be searched to see if it contains the same bin number. If LðlÞ ¼ kb ðuÞ, then the variable cj ðiÞ is set to pðlÞ; if kb ðuÞ is not in the list L, then cj ðiÞ ¼ 0. The projection coefficients are found as cj ¼
N f ðuÞ X
qu ðiÞcj ðiÞ;
(4)
i¼1
where Nf ðuÞ is the number of filled bins for the unknown attractor. IV. ROSSLER CIRCUIT EXPERIMENT
As an illustration of the density method, the projection coefficient vectors are used to detect parameter variations in a Rossler circuit. The Rossler circuit here was originally described in Ref. 22. The circuit was described by the equations dx ¼ 104 ½ 0:05x 0:5y z; dt dy ¼ 104 ½ ax þ 0:128y; dt dz ¼ 104 ½z þ gð xÞ; dt ( ) 0 x
