This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2014.2363775, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00150, Rev.1

1

A Neural Model to Study Sensory Abnormalities and Multisensory Effects in Autism Gerardo Noriega, Member, IEEE

Abstract— Computational modeling plays an increasingly prominent role in complementing critical research in the genetics, neuroscience, and psychology of autism. This paper presents a model that supports the notion that weak central coherence, a processing bias for features and local information, may be responsible for perception abnormalities by failing to "control" sensory issues in autism. The model has a biologicallyplausible architecture based on a self-organizing map. It incorporates temporal information in input stimuli, with emphasis on real auditory signals, and provides a mechanism to model multisensory effects. Through comprehensive simulations the paper studies the effect of a control mechanism (akin to central coherence) in compensating the effects of temporal information in the presentation of stimuli, sensory abnormalities, and crosstalk between domains. The mechanism is successful in balancing out timing effects, basic hypersensitivities and, to a lesser degree, multisensory effects. An analysis of the effect of the control mechanism's onset time on performance, suggests that most of the potential benefits are still attainable even when started rather late in the learning process. This high level of adaptability shown by the neural network highlights the importance of appropriate teaching and intervention throughout the lifetime of persons with autism and other neurological disorders. Index Terms— Autism, self-organizing maps, sensory abnormalities, multisensory integration, weak central coherence.

D

I. INTRODUCTION

copious research and diligent effort in many fronts, regrettably, much about autism remains a mystery. In very general terms, autism is a pervasive developmental disorder of neurological origin, characterized by impairments in the areas of socialization, communication and imagination. Research through the last few decades has yielded important advances in the areas of genetics, neuroanatomy, neurophysiology and psychology. Two comprehensive and well-known references amongst the vast literature on the subject are [1] and [2]. A concise and very accessible introduction to autism is presented in [3]. While remaining behavioural-based, the diagnosis of autism has seen important recent changes. Under the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5), the term Autism Spectrum Disorder (ASD) now ESPITE

Manuscript received xxxxxx; revised xxxxxxx; accepted xxxxxx. The author is with RMS Instruments Ltd., Mississauga, ON L4V 1L9, Canada (e-mail: [email protected]). His work in the general area of neural engineering is carried out on a private capacity.

applies to a single condition with different levels of symptom severity, in contrast to earlier versions of the manual which defined four separate disorders. The related diagnosis of Social (Pragmatic) Communication Disorder is applicable when deficits in the general area of “imagination” (restricted repetitive behaviours, interests and activities) are absent. Much has been written and debated about the merits and potential problems with the new diagnosis criteria (e.g., [4]– [5]). Importantly, and of particular interest in the present work, sensory issues have now been given well-deserved prominence. DSM-5 specifies “hyper- or hyporeactivity to sensory input or unusual interests in sensory aspects of the environment,” as one of at least two “restricted, repetitive patterns of behaviour, interests, or activities” that must be present for diagnosis. Sensory abnormalities refer to the unusual perception and response to sensory stimuli. Typical examples include apparent indifference to pain/temperature, adverse response to specific sounds or textures, excessive smelling or touching of objects, and visual fascination with lights, water or movement. Hyper- and hyposensitivities to sensory inputs, with varying degrees of severity, have been reported clinically and anecdotally for quite some time, but only recently have seen growing interest in research [6]–[8]. While the root cause of sensory abnormalities is not well established, there is considerable evidence that sensory information is processed differently in individuals with autism [1]–[2], and that abnormal cross-modal interactions between sensory systems may be involved. In [8] a cognitive neuroscience theoretical view on multisensory integration is presented, valuable for the study of the perceptual experience of individuals with autism. The work in [9] suggests that some persons with autism have an abnormal cross-modal interaction between the auditory and the somatosensory systems, likely associated with abnormal involvement of the non-classical auditory pathways. The comprehensive review in [10] indicates that children and adolescents with autism have altered intersensory processing profiles when compared with typically developing children. Weak central coherence (WCC) is a cognitive theory first introduced by Frith [2] which hypothesizes that individuals with autism show a processing bias for features and local information, with a relative failure to understand context or meaning. It suggests that the inability to process information to derive meaningful ideas could explain the essential features of autism. Importantly, WCC is unique in addressing both deficits and assets in autism spectrum disorders – for instance,

1534-4320 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2014.2363775, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00150, Rev.1

outstanding mathematical and engineering skills in individuals who struggle with basic language and self-care skills. Numerous studies addressing WCC in autism have been published since [2]. A comprehensive review is available in [6]. The theory has evolved significantly [6], [11] in response to empirical studies that followed its initial introduction. WCC is now viewed as an aspect of cognition in autism, independent of deficits in social cognition. The original view of a core deficit has been replaced by that of a processing bias (which can in fact be overcome in tasks with explicit demands for global processing). The current view of WCC is thus that of a secondary outcome, rather than a fundamental problem, best characterized as a superiority in local processing. WCC remains solidly based on an empirical foundation [6], and is unique in addressing aspects of autism neglected by other accounts. For example, poor generalization skills may be the result of coding experiences in terms of details, rather than extracting prototypes. In the framework of the present work, hyper- and hyposensitivities may be related to a context-free form of processing in autism, that is in contrast to "normal" processing in which context-based interpretation and expectations modulate the experience of sensory inputs. A. Computational models of autism and other disorders Over the last several decades numerous computational models, which to various degrees adhere to constraints of biological plausibility, have been proposed to address forms of cognitive and developmental disorders. Early examples with a fairly generic scope include [12]–[14]. A growing number of papers focus on autism; [15]–[20] are based on the framework of self-organizing maps (SOMs), while [21]–[26] are built on other types of connectionist networks. The work in [27] presents a very comprehensive model showing how autistic behavioural symptoms may arise from prescribed breakdowns in brain processes that involve regions like the prefrontal and temporal cortex, amygdala, hippocampus, and cerebellum. A number of the references cited above are particularly relevant in the context of WCC. In [19] it is suggested that illdeveloped and highly discriminative cortical maps lead to autistic characteristics; cognitive deficits and isolated skills are explained through analogies between computational properties of the maps and WCC. In [23] a connectionist network model shows how coherence deficits may produce the cognitive problems of autistic individuals. [25] uses a simple neural network to show how hyperspecificity may reflect a feature of the neural codes used to represent concepts in the autistic brain. A tendency to represent things in an extremely specific way may predispose autistic individuals to use an excessively conjunctive form of neural coding, which in turn prevents exploiting overlaps that lead to generalization. The work in [15] suggests that excessive inhibition results in the inadequate formation of cortical feature maps and overspecificity of processing. These concepts were further developed in [16]–[17], where familiarity preference in stimuli selection is shown to result in deficient feature maps. A backpropagation network is used in [21] to show that an inadequate number of hidden units may explain deficits in autism – an

2

excessive number of units relative to a task's complexity may lead to rote learning and poor generalization. [20] analyzes various aspects of learning in autism using SOMs. A modified form of the SOM algorithm with lateral neuron interactions is used to model hyper- and hyposensitivities, and a model with explicit rejection of distant neighbors is used to study focus on details rather than the whole. The SOM-based model proposed in [28] lends support to the notion that WCC may be responsible for perception abnormalities by failing to "control" (a) natural variations in sensitivities to sensory inputs, and/or (b) abnormal hyper- and hyposensitivities characteristic of autistic individuals. This model constitutes the foundation of the present work; key points are summarized below for convenience: – Individuals with autism show abnormal hyper- and hyposensitivities to sensory inputs with varying degrees of severity. Normal central coherence may provide a mechanism to compensate for these sensory issues. In the case of severe abnormalities this compensation may result in relatively mild perception issues. When compounded with WCC, however, the outcome would be of severe perception problems. – Individuals outside of the autism spectrum experience natural variations in sensitivity to sensory inputs, as well as in the intensity of the inputs. Inadequate compensation, resulting from WCC, may lead to mild abnormalities in perception. – Thus, different levels of severity in co-existing WCC and sensory abnormalities may result in a "spectrum" of perception problems, in-line with the accepted notion of an autism spectrum disorder. The present work adapts the model of [28] to work with natural sensory signals, with particular emphasis in the auditory domain. The model is enhanced to study the role that multisensory effects may play in the development of the SOMs. This is motivated by the mounting evidence of auditory sensory issues in autism [29]–[31], and of abnormal cross-modal interactions between sensory systems [8]–[10]. To deal with natural input signals in a biologically-plausible manner, further enhancements allow incorporation of temporal information in the presentation of stimuli. B. Biological context The biological basis of the model employed stems from the well-documented functionally-mapped columnar structures of neurons found in the cortex, and similar structures in other areas of the brain. Throughout the auditory system, in fact, the sensory inputs form orderly tonotopic maps of neurons from the cochlear nucleus to the auditory cortex [32]–[33]. Each of the three divisions of the cochlear nucleus (anteroventral, posteroventral and dorsal) is tonotopically organized, with the best frequencies of the neurons forming spatially-ordered maps. The lateral superior olive also shows a tonotopic organization, with a large portion of the nucleus devoted to high frequencies [33]. The central part of the central nucleus of the inferior colliculus (ICC) has a clear laminar structure, with sheets corresponding to isofrequency planes. Adjacent laminae may form functionally discrete modules, with apparently relatively

1534-4320 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2014.2363775, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00150, Rev.1

limited interaction between sheets. Each frequency band lamina is two-dimensional. There may therefore be further functional zones, arranged orthogonal to the frequency axis. Efforts to identify maps, apart from the tonotopic one, have met with limited success; an axis mapping encoding different periodicity, for example, has been suggested [33]. The ventral nucleus of the medial geniculate body, the thalamic relay of the auditory system that projects to the cortex, has a tonotopic organization, with low and high frequencies located dorsally and ventrally, respectively. It is suggested [33] there may be a further level of organization within the nucleus, with functional groups of neurons (‘slabs’) representing ranges of frequencies. In the surface of the auditory cortex there are tonotopically organized fields of neurons, with multiple representations covered by each field. A sound of a specific frequency is represented by a one-dimensional strip of cells on the surface of these fields. Both the visual and the auditory cortex contain functional modules repeated across the surface of the cortex, with each module having a columnar organization, with all cells in one column having related properties. In the visual cortex, for example, line orientation, eye of stimulation and colour are represented by different modules. In the auditory cortex there are different types of structures independent of the frequency-band strips; for example, cells with broad suprathreshold bandwidths are segregated in patches from those with narrow bandwidths [33]. The ubiquitous tonotopic organization is also seen in centrifugal pathways (those running from the higher stages of the auditory system to the lower ones). Fibers in the olivocochlear bundle, arising in the superior olivary complex and innervating the cochlea, show a tonotopic projection structure, frequency-tuned to the region they innervate [33]. SOMs, characterized by a structure of neurons that becomes sensitive to features in input stimuli, provide mappings very well suited to represent the topological structures seen in biological networks. Importantly, the model employed (described in detail in Section II) incorporates lateral interactions between neurons to reinforce or weaken learning. This process is modulated by sensitivity functions that encode sensory sensitivities and multisensory effects. The local computations in the model are in-line with the interactions between neurons through excitatory and inhibitory connections seen in biological systems, controlled by various neurotransmitters and receptors [28]. Sensory sensitivities would of course be only one of many factors involved in determining the manner in which synaptic weights are specified. A simple analogous example is found in the adjustment of sensory thresholds by psychological factors (e.g., the threshold for pain is elevated during competitive sports). The intensity of a stimulus, encoded by the firing frequency of sensory nerves and by the size of the responding receptor population, is effectively modulated along the pathways in sensory systems through presynaptic inhibition or facilitation (i.e., by reduction or increase of the amount of transmitter released). This presynaptic modulation is akin to the lateral neuron interactions in the model employed, and is

3

known to occur typically in early sensory pathways [32]. There is pervasive multisensory convergence along the auditory pathway. This is likely result of the ease with which it can be cross-modally innervated – the auditory brainstem nuclei, for example, lie in the middle of a stream of somatosensory pathways [34]. Somatosensory information, thus, repeatedly enters the auditory pathway at multiple stages, from the cochlea to the cortex. Somatosensory inputs to the cochlear nucleus (CN) convey information from the face and oral structures, and proprioceptive inputs from the trunk, shoulders, head/pinna and neck. The somatosensory inputs to the CN can generate responses with complex excitatory/inhibitory components. Inputs to the dorsal cochlear nucleus (DCN) may contribute to the suppression of sounds from self-generated vocalizations and by movements [33]. Close to 80% of DCN neurons show multisensory integration when stimulated with a combination of auditory and somatosensory inputs [34], with the majority of bimodal neurons showing bimodal suppression. The strength of integration depends on the time interval between stimuli; the integration is apparent with up to around 100 msec between auditory and somatosensory stimuli. The last point highlights the motivation behind encoding timing information in the model developed (Section II). In the inferior colliculus (IC), the external cortex (ICX) receives somatosensory inputs from the dorsal columns and the trigeminal nuclei [33]. The ventral ICX is the primary IC region for processing multisensory information, with overlapping projections from ascending auditory and somatosensory pathways, as well as auditory corticofugal inputs [34]. While the predominant effect of bimodal (auditory and somatosensory) processing is suppression of sound-evoked discharges, trigeminal inputs can also enhance auditory responses to improve signal-to-noise ratio. ICX projects, amongst others, to the superior colliculus (SC). Space maps in ICX likely contribute to joint visualauditory maps in SC. Neurons in SC synthesize information from the various sensory inputs synergistically. Sensory representations are organized topographically, with the different maps sharing the same basic template; thus, neurons in a vertical column of tissue represent the same region of, for example, auditory, visual, and somatosensory space [35]. While the circuitry implementing the integrative function in SC is relatively poorly understood, SC is often used as a physiological model to study multisensory integration [36]. Multisensory convergence continues in the thalamus [37] (where a given thalamic nucleus may receive inputs from a single or multiple modalities, and/or from a source in which two or more modalities converged at earlier stages), and ultimately in low- and high-level cortices [38]–[39]. II. THE MODEL Reference [28] introduces a model of sensory abnormalities in autism and a feedback system to compensate them, based on a measure of global perception. A modified form of the conventional SOM algorithm models sensory abnormalities

1534-4320 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2014.2363775, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00150, Rev.1

through lateral interactions between neurons. The sensory abnormalities are described through sensitivity functions, and input stimuli to the SOM are passed through a stimuli filter. A feedback system calculates a measure of balance of "global" performance of the SOM over multiple sensory domains, and generates control variables used to tune the performance of the stimuli filter. The model was implemented with synthetic stimuli randomly distributed between four domains. The present work uses as a core the model of [28]. It enhances two fundamental aspects of it. Firstly, while in [28] the sensitivity functions where used only to describe sensitivity abnormalities (assumed known a priori), the present work uses these functions to model both, sensitivity abnormalities, and multisensory interactions. This provides a useful mechanism to model crosstalk between the different sensory domains. Secondly, the time dimension is introduced into the model by presenting stimuli to the network in the sequential order in which input samples where acquired. Note that this timing is independent of (and at a finer scale than) the epochs associated with the SOM learning process. This timing element allows the analysis of SOM performance when processing natural signals such as, for example, speech or music, in which time plays a critical role. Precedence in models that integrate temporal information into the SOM to address dynamic sensory processing can be found in [40]. The model proposed in [41] studies how propagation delays resulting from abnormal brain overgrowth in early developmental stages of children with autism, may be conducive to poorly structured feature maps. Without loss of generality, the focus of the present work is on the auditory domain. This is motivated by the strong evidence of its connection with autism discussed in Section I. Two-dimensional time-frequency images are commonly used representations for sound/speech analysis. The cochleagram (or spectrogram) plots cochlear model estimates of auditory nerve firing probabilities as a function of time. Auditory nerves are stimulated at specific cochlear locations. Fibers of the auditory nerve have lower thresholds to tones of certain frequencies; tuning curves relate threshold and stimuli frequency, and have a threshold minimum at the characteristic frequency [33]. To represent the auditory stimuli, in a first step the Lyon's passive long-wave cochlear model [42], based on a transmission line model of the basilar membrane followed by several stages of adaptation, is used to generate a cochleagram of a speech signal. In a second step a two-dimensional stimuli map is generated from the cochleagram, with cochlear place (i.e., frequency) mapped along one dimension, and firing rate along the other. It is the stimuli map that is used as input to the SOM-based model, with temporal information preserved by presenting stimuli to the SOM in sequential order. The biological motivation behind this representation lies fundamentally in the tonotopical structure encountered throughout the auditory system, starting at the cochlea, upstream through the cochlear nucleus and all the way to the auditory cortex (Section I.B). In addition to the auditory domain, the model represents three additional sensory domains in an analogous manner, but

4

using synthetic data. The two dimensions of the corresponding stimuli maps could have various interpretations depending on the type of domain represented (e.g., two-dimensional visual maps); temporal information is encoded in the order of presentation of the stimuli, as is done with the auditory domain. The use of synthetic data for these domains facilitates experimentation in a controlled fashion. The block diagram in Fig. 1 illustrates the model. The stimuli from the four stimuli maps are presented to their respective domains in each of four quadrants of the twodimensional SOM network. For each epoch throughout learning, all stimuli sampled at discrete times tj, j = 0, 1, ..., Nt–1, are first randomized and then presented to the network. The sensitivity functions model sensitivity abnormalities, typically in the form of hyper- or hyposensitivities in certain domains, and multisensory effects. The general form for the sensitivity function for the d-th domain, Sd, is as follows:

S d (e, j ) =

∑ g m ,d [ ym ( j ), e] + α d (e )

∀m ≠ d

(1)

where e is the epoch in the learning process, and j represents discrete time (i.e., tj = jT, with T the sampling period of the original signals). The functions gm,d model multisensory effects of domain ‘m’ on domain ‘d’; they are a function of both the input signal from domain ‘m’, ym (a function of time), and the epoch number. The αd model sensory abnormalities; they are dependent only on epoch number, as hyper- and hyposensitivities will vary only on a coarse time scale. The rest of the model is described in detail in [28]. In what follows we provide only a brief summary. The SOM is a competitive neural network that performs a mapping of a q-dimensional input space into an l-dimensional feature space [43]–[45]. At each iteration k, neurons in the network (i = 1, 2, …, NN) are characterized by their weight vector wi(k) = [wi1(k) wi2(k) … wiq(k)] , and a position in an ldimensional lattice given by the vector vi(k) = [vi1(k) vi2(k) … vil(k)]. The position of the winning neuron, vWN(k), is determined by comparing the input vector (stimulus) x(k) = [x1(k) x2(k) … xq(k)] to the weight vectors of all the neurons: min { |x(k) – wi(k)| } = |x(k) – wWN(k)|, over i = 1, 2, …, NN. (2) The weights of neurons in the neighborhood of the winning neuron are adjusted with a strength N(yi, k) proportional to their distance to the winning neuron, yi(k) = |vi(k) – vWN(k)| , for i = 1, 2, …, NN.

(3)

In a first step, the algorithm adjusts weights per (4): wi(k+1) = wi(k) + η(k) N(yi, k) [x(k) – wi(k)] .

(4)

The neighborhood function N(yi, k) is typically Gaussian, with standard deviation σ(k). Both σ(k) and the algorithm’s learning rate, η(k), usually decrease exponentially during the ordering phase (EORD epochs long), and then linearly in the convergence phase (ECVG epochs long). In a second step neuron weights are adjusted toward the winning neuron, proportionally to the sensitivity function for the domain in which the stimulus is located: (5) w i (k ) = wˆ i (k ) + [wˆ WN (k ) − wˆ i (k )]S d (e, k ) .

1534-4320 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2014.2363775, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00150, Rev.1

5

Stimuli Maps

Cochleagram (-like) Maps

δ ij = Cij / N S Input Stimuli

Multisensory Effects (1)

Time Sequencing (6)

Sensory Abnormalities (1)

Sd Stimuli Filter

Modified-SOM (2)-(5)

wi(k) [i=1,...,NN]

p d (bd , β ) [Fig.2]

Control Variables

Feedback System

β

Region Coverage, c d Balance Index (7a)-(7c)

(8)

In-bounds ratio, bd (7d)

Fig. 1. Block diagram of the model. Subscript ‘d’ denotes the domain number. The panel at the output of the ‘Modified-SOM’ block shows an example of the SOM at end of learning, under baseline conditions, per Case A3 (with domains numbered clockwise, starting at the upper-right quadrant).

In (5), wi(k) is the updated weight for the i-th neuron at iteration k, wˆ WN (k ) and wˆ i (k ) are the weights of the winning neuron and the i-th neuron, respectively, after adjustment using (4), Sd(e,k) is the sensitivity function for domain d, and e denotes the epoch number. For each epoch throughout the learning process the algorithm (2)–(5) iterates over all active input stimuli. The stimuli are ordered chronologically in groups, and then randomized within the groups: {M0 stimuli at time t0} ∈ Group 0: k = 1 : M0 {M1 stimuli at time t1} ∈ Group 1: k = M0+1 : M0+M1 ... {MNt–1 stimuli at time tNt – 1} ∈ Group Nt–1: (6) k = M0+M1+...+ MNt–2+1 : M0+M1+...+ MNt–1 A. Basic performance measures Performance measures suitable for the context of the present work were introduced in [20]. In what follows, appropriately adapted versions of the domains’ coverage c, and in-bounds ratio b are used. The coverage of a domain is a measure of the effectiveness with which the stimuli distributed over a domain are mapped by the subset of neurons that fall within it. The model is implemented on a 2-dimensional input space (q=2) and a 2dimensional feature space (l=2), with a total of NN neurons in an NR x NC network. The input space is divided into four identical quadrants, corresponding to the four sensory domains considered. For a given domain, the stimuli density is first calculated for each of the cells in a rectangular mesh, NR/2 x NC/2, over the domain area:

(7a)

where Cij is the number of stimuli within cell (i, j), and NS is the number of stimuli addressed to the domain. To achieve coverage commensurate with stimuli density, the expected number of neurons in each cell would be (7b) nij = δ ij N N / 4 . The coverage for the domain is then defined as (7c) c = N COV / N ACT , where NCOV is the number of cells with sufficient coverage (i.e., the number of neurons in the cell is at least nij), and NACT is the number of active cells (cells with at least one stimulus). The in-bounds ratio is a measure of the number of neurons that fall within the overall span of a given domain, regardless of the effectiveness of the distribution of the neurons: (7d) b = I / (N N / 4) , where I is the number of in-bound neurons (i.e., the total number of neurons in active cells). It is straightforward to see that 0 ≤ c ≤ 1 (with c = 0 representing null coverage, and c = 1 full coverage), and 0 ≤ b ≤ 4 (with b ≈ 1 in normally developed maps, b < 1 in undercovered domains, and b > 1 in over-covered domains). B. The feedback system The feedback system establishes a measure of the balance achieved by the SOM over the four domains considered, and is used to calculate control variables which configure the stimuli filter. A network which provides roughly equivalent coverage for all four domains will yield a good measure of balance; this is akin to good global perception. A lack of balance in coverage is seen as a preference or bias for local information. The measure of balance tunes the stimuli filter; stimuli targeted to well-performing domains are filtered out with an increasing probability as the unbalance increases. The feedback system operates on a coarse time basis, updating variables every epoch. The region balance index β is calculated by adding the differences between coverages of all pairs of domains: 4

4

β = ∑ ∑ ci − c j . i =1 j = i +1

(8)

With 0 ≤ ci ≤ 1, it is clear that 0 ≤ β ≤ 4, with β = 0 corresponding to a perfectly balanced network (optimal global perception), and the measure increasing up to a maximum value of 4 as the balance deteriorates. Note that above, and in what follows, for simplicity the notation does not explicitly show the dependence on the epoch number. The feedback system calculates a set of variables pd(bd, β), which control the operation of the stimuli filter. The variables represent the probability with which input stimuli to domain d will be accepted by the filter. The function pd(bd, β) captures effectively conditions indicative of sensory abnormalities, and a measure of the balance of coverage (global perception) achieved over the various domains. For brevity, we only show pd(bd, β) graphically in Fig. 2, configured with the parameters used throughout the present work. Its explicit form and a detailed description of the parameters available to configure it may be found in [28].

1534-4320 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2014.2363775, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00150, Rev.1

Fig. 2. Function pd(bd, β) defines the probability of acceptance of stimuli for a given domain d, as a function of the in-bounds ratio for the domain bd and the region balance index β.

III. SIMULATIONS, ANALYSIS AND RESULTS A. The effect of timing in the presentation of stimuli The first aspect investigated is whether preserving the time dimension in the order of presentation of the stimuli has an effect in the development of the SOMs. The input to the to the auditory domain (D1) is derived from the cochleagram of the speech signal of a child saying the words "little bear". The audio signal spans for a total of 2.75 seconds and was sampled at 16 kHz. The Lyon's passive cochlear model [42] was used with a decimation factor of 100 (with suitable antialiasing). To eliminate background noise a threshold at 0.3 intensity was applied. The top panels in Fig. 3 show both the cochleagram and the corresponding stimuli map. The three-syllable utterance yields a fairly rich frequency content from the harmonics of the vowel sounds, and well defined sharp onsets for the laminal-dental “t” and the labial “b” consonant sounds. For the three additional domains, synthetic signals were generated as shown in the panels of the second row (D2), third row (D3), and fourth row (D4) in Fig. 3. For simplicity, the vertical axes in the cochleagram-like mappings for domains D2–D4 are interpreted as per the auditory domain, i.e., in units of 'cochlear place' (or frequency); suitable scaling would be in place if a specific sensory domain was considered. Similarly, the signal intensity is simply normalized to unity. For domain D2 the signal steps through the full range of "frequencies", from the lowest (1) to the highest (86), in short bursts of approximately 187.5 msec. Each burst includes a range of frequencies in three bands; the low- and high-band contain low signal intensities (uniformly distributed in the range 0.3–0.5), while the mid-band contains high intensities (uniformly distributed in the range 0.8–1.0). In D3 all stimuli are concentrated into one burst, starting at around 25% into the record, and with duration of approximately half the record. The burst covers a narrow frequency band (40–53), with the intensity increasing linearly with time, from 0.0 to 0.45 in the first half of the record, and then from 0.55 to 1.0 in the second half. In D4 all stimuli are concentrated into two short bursts (slightly over 200 msec each), one starting at around 25% into the record, and the other 75% into it. The bursts cover fairly large frequency bands (20–60 and 41–81), with some overlap. Intensities are uniformly distributed in the range 0.2–0.5 in the

6

first burst, and in the range 0.4–0.7 in the second burst. An important aspect of the domains for the purpose of the analysis in this section is the similarity between the auditory domain (D1) and D4, in that their stimuli are concentrated in a few bursts of short duration. This is in contrast to D2 and D3, in which the stimuli are spread over time. Note also that the stimuli layout for the synthetic domains was devised so as to demand precise positioning of the neurons to achieve a good measure of coverage; this helps emphasize differences in performance under the effect of the conditions investigated – note the multiple narrow frequency bands in the stimuli map for D2, the narrow gap between the two intensity ranges covered in D3, and the higher stimuli density area in D4, resulting from overlaps both in frequency and time. Simulations were conducted with three different general configurations: 1) stimuli presented in random order, with no control mechanism in place; 2) stimuli presented in sequential order, with no control mechanism in place; 3) stimuli presented in sequential order, with a control mechanism in place. In all three cases (referred to as A1, A2, and A3 in what follows) the sensitivity functions for all four domains were disabled, i.e., S d (e, j ) = 0 , for d = 1, ..., 4. The SOM network was configured with NR = 40 rows and NC = 40 columns (which keeps computational demands at a reasonable level, without loss of generality), EORD = 250 epochs, and ECVG = 100 epochs. The modified SOM algorithm is configured with a neighborhood function of Gaussian type, NG(y, i) = e − y / 2σ (i ) , where i is the epoch number, and y the distance to the winning neuron. The standard deviation decreases exponentially during the ordering phase, σ (i ) = (σ in − σ ∞ )e − k i + σ ∞ , and linearly in the convergence 2

2

σ

phase, σ (i ) = (σ end − σ o )i / E CVG + σ o . The algorithm's learning rate behaves similarly, with parameters ηin, η∞, ηend and kη. Throughout the simulations the algorithm was configured with σin = 6, σ∞ = 0, σend = 0.1, kσ = 0.0072 (so that σ(250) = 1), ηin = 0.01, η∞ = 0, ηend = 0.001 and kη = 8.9257x10-4 (so that η(250) = 0.008). This yields baseline performance with inbounds ratios close to 1 and coverage around 0.8 by the end of the learning process. Where applicable, the control mechanism of Fig. 2 was used (Control II in [28]). A total of 15 runs were carried out with each of the three configurations. For baseline conditions (Case A1, temporal information excluded in presentation of stimuli) all coverage and in-bounds ratio measures evolve similarly, ending at values around 0.8 and 0.8–1.0, respectively, at the end of learning. The top panels in Fig. 4 illustrate coverage performance, relative to baseline, for Case A2 (left – temporal information included, no control mechanism), and Case A3 (right – temporal information included, control mechanism in place). The ratios plotted are averages over the 15 runs. For Case A2 performance for D2 and D3 (around 0.95 and 1.2 relative to baseline, at e = 350; Fig. 4, top-left panel) is clearly superior to that of D1 and D4 (around 0.8 and 0.85 relative to baseline). For Case A3, when the control mechanism is in place, for the most part performance can be seen to roughly return to the original baseline levels. The behaviour observed

1534-4320 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2014.2363775, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00150, Rev.1

7

for the in-bounds ratio measures is in general similar to the one for the coverage. The above suggests that (i) without the control mechanism in place, the domains whose stimuli are concentrated into short-duration bursts, when other domains have stimuli input spread out throughout the duration of the record, will see a deterioration in performance; (ii) the control mechanism effectively balances out the timing effects.

(a)

(b)

(c)

(d) Fig. 3. Input stimuli for cases A1–A3: (a) Cochleagram (left) and stimuli map (right) for input to auditory domain D1; (b)–(d) Cochleagram-like mappings (left) and stimuli maps (right) for synthetic inputs to D2–D4. For cases B1– B3, input to D1 is per (a), input to D2 per (d), and inputs to D3 and D4 are similar to (d), but with shifts in time, frequency, and intensity (Section III.B). Horizontal axes in left panels scaled in ‘samples’; time=samples/160 [sec].

Table I (top) shows the ANOVA for the coverage (c1–c4), the in-bounds ratio (b1–b4), and the regional balance index (β), for cases A1–A3, taken in pairs, evaluated at the end of the ordering phase (e=250) and at the end of the convergence phase (e=350). The effect of the temporal information is clearly statistically significant (p < 0.05) on all the measures considered (A1 vs A2 in Table I), from as early as e=250. The effectiveness of the control mechanism in compensating for these effects is very evident for domains D2 and D4 (differences are not statistically significant at p < 0.05 – see A1 vs A3, Table I), but more subtle for domains D1 and D3 (differences remain significant at p < 0.05). B. Multisensory effects The focus now shifts to the effects that interaction between the different sensory domains may have on the development of

the SOMs. The input to the auditory domain (D1) remains the same as in Section III.A (Fig. 3, first-row panels). Synthetic signals are once again generated for D2–D4. The input to D2 is the same that was used in Section III.A for domain D4, with stimuli concentrated into two short bursts; it is thus of similar nature to the input to the auditory domain with respect to timing (Fig. 3, fourth-row panels). To maintain the timing aspect of all inputs analogous, the input signals for D3 and D4 are similar to the input to D2, with the following differences: (i) the locations in time of the bursts of stimuli are different (non-overlapping); (ii) the frequency ranges covered are shifted down by 5 units (D3) and 10 units (D4); and (iii) intensity ranges are shifted up by 0.1 units (D3) and 0.2 units (D4). In all simulations described in this section stimuli were presented observing the timing of the signals' samples. Simulations were conducted with three different general configurations: 1) no sensory abnormalities, no multisensory effects, and no feedback control mechanism in place; 2) a hypersensitivity and crosstalk effects on D1, with no control mechanism in place; 3) a hypersensitivity and crosstalk effects on D1, with a control mechanism in place. The three cases are referred to as B1, B2, and B3 in what follows. The SOM network was configured with NR = 40 rows and NC = 40 columns, EORD = 300 epochs, and ECVG = 100 epochs. The modified SOM algorithm was configured in a manner similar to that of Section III.A, but with parameters σin = 6, σ∞ = 0, σend = 0.1, kσ = 0.0060 (so that σ(300) = 1), ηin = 0.01, η∞ = 0, ηend = 0.001 and kη = 7.4381x10-4 (so that η(300) = 0.008). With these settings the model yields baseline performance with in-bounds ratios close to 1 and coverage around 0.8 by the end of the learning process. Where applicable, the control mechanism of Fig. 2 was used (Control II in [28]). The sensitivity function for the auditory domain includes: a. A simple hypersensitivity modeled by an exponential decay toward zero. This form is consistent with outputs generated by first-order dynamical processes. Typically, a hypersensitivity of this type will improve performance in the hypersensitive domain relative to the others. If it is very intense, however, it may in fact result in poorer coverage for the hypersensitive domain, as most neurons converge into a small area [28]. In the context of autism, hypersensitivities are in some cases conducive to outstanding performance in certain areas (e.g., math, software engineering, meticulous attention to detail); when some threshold level is exceeded, however, they may lead to such acute levels of specialization that all other sensory input is practically dismissed. b. Multisensory effects modeled as “crosstalk” from the other domains. The crosstalk is proportional to the average intensity level of all active stimuli (over D2, D3 and D4) at each sample point. This is in-line with well documented accounts of sensory “overload” issues in autism [29]–[30]. Thus, the sensitivity function (1) for D1 takes the form: S1 (e, j ) =

K0 N2 + N3 + N4

N3 N4  N2   ∑ y 2,k ( j ) + ∑ y3,k ( j ) + ∑ y 4,k ( j ) + K1 exp(− K 2 e ) k =1 k =1  k =1 

(9) where e is the epoch number, j is discrete time, and N2, N3 and N4 are the number of active stimuli at time j, for domains D2,

1534-4320 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2014.2363775, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00150, Rev.1

D3 and D4, respectively. The functions yd,k(j) represent the intensity of the k-th stimulus on domain d at time j. Note that the multisensory effects are modeled at a fine time scale level (varying with discrete time), whereas the hypersensitivity is modeled at a coarse time scale (varying with epoch number). For the simulations below the constants in (9) where set to: K0 = 1x10-5, K1 = 5x10-5, K2 = 0.018. The sensitivity functions for D2–D4 were disabled, S d (e, j ) = 0 , for d = 2, 3, 4. As before, 15 runs were carried out for each of the three cases considered. For baseline conditions (Case B1, no sensory abnormalities, no multisensory effects, no control mechanism), the coverage and in-bounds ratio for all domains are quite similar throughout all epochs, the former ending roughly in the range 0.75–0.85, and the latter in the range 0.9– 1.0 at e = 400. The bottom panels in Fig. 4 illustrate coverage performance, relative to baseline, for Case B2 (left – hypersensitivity and crosstalk in D1, no control mechanism), and Case B3 (right – hypersensitivity and crosstalk in D1, control mechanism in place). The ratios plotted are averages over the 15 runs. The hypersensitivity and multisensory effects (Case B2) are very evident in the coverage (as well as in the in-bounds ratio, not shown for brevity). In the auditory domain both measures remain (well) above baseline throughout learning, at the expense of the other domains for which they are for the most part well below baseline, rapidly deteriorating beyond e ≈ 200. With the feedback mechanism in place (Case B3), the performance gap between the auditory domain and the others is clearly reduced. Up to around the end of the ordering phase (e = 300) coverage for the four domains is quite close to baseline (within roughly +10% / –15%). Beyond this point, the measures diverge with c1 remaining fairly stable, while c2–c4 decrease. This reflects the lingering crosstalk effects, which unlike the hypersensitivity, do not diminish with the epochs (9). The control mechanism appears to do well in compensating for the hypersensitivity (in accordance with [28]), but yields more modest results in dealing with the multisensory issues. Table I (bottom) shows the ANOVA for the coverage (c1– c4), the in-bounds ratio (b1–b4), and the region balance index (β) measures, for cases B1–B3, taken in pairs, evaluated at the end of the ordering phase (e = 300) and at the end of the convergence phase (e = 400). Performance at end of learning when the hypersensitivity and multisensory effects are present, without control, is significantly different from baseline (B1 vs B2). The corrections brought in by the feedback mechanism are also statistically significant (B2 vs B3), with the exception of the coverage for D1; while c1 remains fairly unchanged throughout the epochs, the bulk of the control effects is seen in the improvements to c2–c4 (see also Fig. 4). While the control mechanism significantly improves performance, this remains by enlarge significantly different from baseline (B1 vs B3); note (Table I) that at e = 300, before coverage measures begin to diverge, the region balance index, β, for the controlled case (B3) does not differ significantly from baseline (B1).

8

Fig. 4. Coverage performance; averages of 15 runs. Top-left: Case A2 (temporal information, no control) relative to baseline (Case A1, no temporal information, no control). Top-right: Case A3 (temporal information, control) relative to baseline (Case A1). Bottom-left: Case B2 (hypersensitivity in D1, multisensory effects, no control) relative to baseline (Case B1, no sensory abnormalities, no multisensory effects, no control). Bottom-right: Case B3 (hypersensitivity in D1, multisensory effects, control) relative to baseline.

C. Time of onset of the control mechanism The question posed in this section deals with the effect of the control mechanism’s onset time on the development of the neural maps. In other words, how effective is the feedback control mechanism if started “late” into the learning process? In as much as the feedback system may be akin to a mechanism conducive to sound global perception [28], this parallels crucial questions on the importance of early intervention in autism through highly structured and intensive teaching methods, such as those based on the principles of applied behaviour analysis – see, for example, [46]. The model and input stimuli are per Case B3, Section III.B (a hypersensitivity and crosstalk effects are active on D1), but the crosstalk level is significantly higher, with K0=1x10-4 in (9). Baseline performance is with the control mechanism (Fig. 2) in operation from the beginning of learning. In Table II performance is evaluated at the end of learning (e = 400), relative to baseline, for different onset times for the control: no control (equivalent to Case B2, Section III.B), control starting at 75%, 50% and 25% into the learning process. The results shown are averages over 15 runs. Interestingly, in terms of coverage performance is within 15% of baseline for all domains, even when the control mechanism is started as late as 75% into the learning process. This is particularly relevant for the domains without a hypersensitivity or multisensory effects (D2–D4), which have rather poor performance when the control is off (e.g., c3 = 0.08). The in-bounds ratios b2–b4 are roughly within 20% of baseline with the control started as late as 75% into the record; b1, on the other hand, needs the control started earlier (at 50%) to yield performance close to baseline (within 5%). The ANOVA shows there is in fact no statistically significant difference in any of the measures with the control started at 50% into the record or earlier. For c1 and c2, this is also the

1534-4320 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2014.2363775, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00150, Rev.1

9

case with the control started as late as 75% into the record. Encouragingly, thus, it appears that the network remains flexible enough to adjust fairly rapidly once the control mechanism is activated. Most of the benefits of the control can

be attained even if started late in the learning process. Early onset times seem to benefit the most domains subjected to sensory abnormalities and/or multisensory effects.

TABLE I ANOVA FOR COVERAGE (c1–c4), IN-BOUNDS RATIO (b1–b4), AND REGIONAL BALANCE INDEX (β). TOP: CASES A1, A2 AND A3 (SECTION III.A). BOTTOM: CASES B1, B2 AND B3 (SECTION III.B). CASES TAKEN IN PAIRS, EVALUATED AT THE END OF ORDERING AND CONVERGENCE PHASES. THREE CASES (a = 3) COMPARED, WITH n = 15 RUNS PER CASE. LSD METHOD [49] COMPARISON OF PAIRS OF TREATMENT MEANS. ENTRIES ARE ‘MEAN DIFFERENCE’ FOLLOWED BY (LSD). LSD = tα/2,N-a(2MSE/n)1/2, WITH α = 0.05, N = n.a = 45, AND MSE THE MEAN SQUARE ERROR. Cases A1 vs A2 e = 250 e = 350 c1 0.133 (0.021) * 0.153 (0.023) * c2 0.053 (0.017) * 0.047 (0.022) * c3 0.152 (0.037) * 0.141 (0.036) * c4 0.130 (0.043) * 0.128 (0.040) * b1 0.234 (0.038) * 0.237 (0.035) * b2 0.056 (0.041) * 0.053 (0.039) * b3 0.460 (0.079) * 0.472 (0.081) * b4 0.172 (0.068) * 0.168 (0.067) * β 0.773 (0.099) * 0.771 (0.093) * Cases B1 vs B2 e = 300 e = 400 c1 0.074 (0.021) * 0.023 (0.017) * c2 0.058 (0.014) * 0.265 (0.013) * c3 0.154 (0.011) * 0.343 (0.011) * c4 0.131 (0.026) * 0.382 (0.024) * b1 0.305 (0.047) * 0.388 (0.041) * b2 0.013 (0.022) 0.178 (0.025) * b3 0.168 (0.019) * 0.348 (0.014) * b4 0.163 (0.049) * 0.414 (0.038) * β 0.242 (0.081) * 0.747 (0.062) * * p < 0.05, tα/2,42 = 2.018

0.024 0.006 0.073 0.042 0.072 0.009 0.120 0.059 0.149 0.070 0.061 0.116 0.102 0.234 0.037 0.115 0.115 0.073

Cases A1 vs A3 e = 250 e = 350 (0.021) * 0.031 (0.023) * (0.017) 0.014 (0.022) (0.037) * 0.065 (0.036) * (0.043) 0.030 (0.040) (0.038) * 0.081 (0.035) * (0.041) 0.000 (0.039) (0.079) * 0.114 (0.081) * (0.068) 0.051 (0.067) (0.099) * 0.094 (0.093) * Cases B1 vs B3 e = 300 e = 400 (0.021) * 0.033 (0.017) * (0.014) * 0.181 (0.013) * (0.011) * 0.237 (0.011) * (0.026) * 0.262 (0.024) * (0.047) * 0.300 (0.041) * (0.022) * 0.120 (0.025) * (0.019) * 0.202 (0.014) * (0.049) * 0.233 (0.038) * (0.081) 0.405 (0.062) *

IV. DISCUSSION The proposed model complements earlier work which supports the notion that WCC may be responsible for perception abnormalities by failing to control natural variations in sensitivities to sensory inputs, and/or abnormal hyper- and hyposensitivities characteristic of autism. This inference is based on the atypical topologies of the SOMs that result with a faulty (or absent) feedback mechanism, based on a measure of the overall balance of stimuli coverage over sensory domains (i.e., “global perception”). Cortical maps with well-structured topologies have important advantages in neural information processing – the mapping of related stimuli in the same area yields shorter communication paths, which in turn yields more robust signal transmission. Furthermore, the clustering of stimuli features in cortical maps may also simplify interactions between cortical areas, facilitating earlystages of development of abstraction and categorization abilities. Poorly-structured maps may result in deficits of varying degrees of severity in the areas just mentioned. The model is designed to work with natural sensory signals, with particular emphasis in the auditory domain, and incorporates temporal information in the presentation of stimuli. Its structure allows simulations to study the role of multisensory effects in the development of the SOMs. The motivation behind the model’s structure lies in the mounting evidence of auditory sensory issues, and of abnormal crossmodal interactions between sensory systems found in autism. The simulations carried out demonstrate that the development of the SOMs is indeed affected when temporal

0.109 0.047 0.079 0.088 0.162 0.066 0.340 0.113 0.624 0.004 0.004 0.038 0.029 0.070 0.024 0.053 0.048 0.169

Cases A2 vs A3 e = 250 e = 350 (0.021) * 0.122 (0.023) * (0.017) * 0.033 (0.022) * (0.037) * 0.076 (0.036) * (0.043) * 0.098 (0.040) * (0.038) * 0.156 (0.035) * (0.041) * 0.053 (0.039) * (0.079) * 0.358 (0.081) * (0.068) * 0.117 (0.067) * (0.099) * 0.677 (0.093) * Cases B2 vs B3 e = 300 e = 400 (0.021) 0.010 (0.017) (0.014) 0.083 (0.013) * (0.011) * 0.106 (0.011) * (0.026) * 0.120 (0.024) * (0.047) * 0.087 (0.041) * (0.022) * 0.058 (0.025) * (0.019) * 0.146 (0.014) * (0.049) 0.181 (0.038) * (0.081) * 0.342 (0.062) *

information is taken into account, and that the control mechanism of the model is overall successful in balancing out timing effects, as well as basic hypersensitivities and multisensory effects. The work further suggests that most of the potential benefits of the feedback mechanism are still attainable even when started rather late in the learning process. The flexibility of the neural network demonstrated by the last point, is especially relevant in highlighting the importance of appropriate teaching and intervention throughout the lifetime of persons with autism and other neurological disorders. TABLE II PERFORMANCE AT THE END OF LEARNING, RELATIVE TO BASELINE, FOR DIFFERENT CONTROL ONSET TIMES. RESULTS ARE AVERAGES OVER 15 RUNS. MODEL PER CASE B3 (SECTION III.B), WITH K0=1X10-4 IN (9). BASELINE: CONTROL IN PLACE FROM THE OUTSET. STATISTICAL SIGNIFICANCE PER ANOVA; EACH CASE VS BASELINE (a = 2), WITH n = 15 RUNS. LSD METHOD [49] COMPARISON OF PAIRS OF TREATMENT MEANS. LSD = tα/2,N-a(2MSE/n)1/2, WITH α = 0.05 OR 0.0005, N = n.a = 30, AND MSE THE MEAN SQUARE ERROR. Coverage c1 c2 c3 No Control: 0.6920 ** 0.1451 ** 0.0895 ** From 75% on: 0.9895 0.9946 0.9102 ** From 50% on: 0.9874 0.9987 1.0165 From 25% on: 1.0050 1.0124 1.0081 In-bounds ratio b1 b2 b3 No Control: 1.7526 ** 0.1133 ** 0.0935 ** From 75% on: 1.3723 ** 0.8832 ** 0.8839 ** From 50% on: 1.0266 0.9877 1.0156 From 25% on: 1.0144 1.0066 1.0140 * p < 0.05, tα/2,28 = 2.048 ** p < 0.0005, tα/2,28 = 3.935

c4 0.1228 ** 1.1652 * 1.0091 1.0228 b4 0.1245 ** 1.1959 ** 1.0122 1.0153

The simple two-dimensional mapping provided by the SOM yields a biologically plausible model of cortical and other feature maps in the brain. With suitable adaptation of the performance measures (7)–(8), the model can be scaled up to

1534-4320 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2014.2363775, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00150, Rev.1

consider more complex multidimensionalities in the sensory inputs. Future work may also involve use of growing hierarchical SOMs, experiments with more complex forms of sensory abnormalities and multisensory effects, and with natural inputs for the non-auditory domains. The two-dimensional stimuli maps for the auditory domain represent frequency (cochlear place) and intensity (firing rate), and are derived from conventional cochleagrams. For the nonauditory domains the two dimensions in the stimuli maps may have various interpretations. For the visual domain, for example, the stimuli maps could correspond to retinotopic maps (two-dimensional neural maps of the visual field in the primary visual cortex and higher-order areas), or perhaps encode other aspects of the visual image such as contrast, spatial frequency, temporal frequency, or form. Stimuli maps for the somatosensory domain could represent location on the skin and intensity for a specific modality (touch, pressure, temperature or pain). The corresponding cochleagram-like maps for the non-auditory domains would not necessarily constitute any conventional representation of the inputs, but nonetheless the biological analogue of the two-dimensional stimuli map representations remains sound (Section I.B). For all domains in the model, temporal information is encoded into the order of presentation of the stimuli. Timing plays a critical role when studying multisensory integration, as responses are affected by the temporal relations between the different stimuli; this constitutes the basis of the so-called temporal principle of multisensory integration [47]. An alternative approach that may yield interesting comparisons would be an architecture using multiple SOMs, one for each domain, that interact with each other. This would be in-line with the view prevalent around a decade ago, consisting of fundamentally unimodal perceptual processing based on hierarchically-connected specialized cortical areas, with multisensory interactions occurring only near the top levels of the hierarchy [48]. This view, however, has changed dramatically over the last number of years. Multisensory research, supported by anatomical and electrophysiological studies, has established that multisensory interactions occur even in primary cortical areas; the cerebral cortex is now viewed as essentially multisensory – practically all cortical areas are influenced by information from multiple senses [48]. The architecture implemented in the proposed model, with a single SOM processing multiple sensory inputs, appears best suited to this current view. Clearly, the processing of sensory information and multisensory integration in the brain are extremely complex. The underlying mechanisms that may lead to neurological disorders such as autism are still not fully understood. Continuing efforts in the area of computational modeling offer a complement to critical research in the areas of genetics, neuroanatomy, neurophysiology and psychology. REFERENCES [1] [2] [3]

F.R. Volkmar, R. Paul, A. Klin and D. Cohen (Eds.), Handbook of Autism and Pervasive Developmental Disorders, 3rd Ed., Wiley, 2005. U. Frith, Autism: explaining the enigma, Blackwell Pub., Inc., 1989. U. Frith, Autism – a very short introduction, Oxford Univ. Press, 2008.

10

[4] [5] [6] [7] [8] [9] [10]

[11] [12] [13] [14] [15] [16]

[17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30]

A. Taheri, A. Perry, “Exploring the proposed DSM-5 criteria in a clinical sample,” J. Autism Dev. Disorders, Vol.42, pp.1810–1817,2012. E.R. Ritvo, “Postponing the proposed changes in DSM 5 for autistic spectum disorder until new scientific evidence adequately supports them,” J. Autism and Dev. Disorders, Vol.42, pp.2021–2022, 2012. F. Happe and U. Frith, “The weak coherence account: detail-focused cognitive style in autism spectrum disorders,” J. Autism Dev. Disorders, Vol.36, No.1, pp.5–25, 2006. O. Bogdashina, Sensory perceptual issues in autism and asperger syndrome: Different sensory experiences – different perceptual worlds, Jessica Kingsley Publishers, London, 2003. G. Iarocci and J. McDoland, “Sensory integration and the perceptual experience of persons with autism,” J. Autism Dev. Disorders, Vol.36, No.1, pp.77–90, 2006. A.R. Møller, J.K. Kern, B.Grannemann, “Are the non-classical pathways involved in autism and PDD?”, Neurol Res, 27(6), pp.625–629, 2005. L.E. Bahrick and J.T. Todd, “Multisensory processing in autism spectrum disorders: intersensory processing disturbance as a basis for atypical development,” in B.E. Stein (Ed.), The new handbook of multisensory processing, MIT Press, 2012. S. Dakin and U. Frith, “Vagaries of visual perception in autism,” Neuron, Vol.48, pp.497–507, November 2005. M. Spitzer, “A neurocomputational approach to delusions,” Compr. Psychiatry, Vol.36, pp.83–105, 1995. A. Oliver, M.H. Johnson and B. Pennington, “Deviation in the emergence of representations: a neuroconstructivist framework for analyzing developmental disorders,” Dev. Science, 3, pp.1–23, 2000. J.A. Reggia, E. Ruppin and R.S. Berndt (Eds.), Neural modeling of brain and cognitive disorders, World Scientific, 1996. L. Gustafsson, “Inadequate cortical feature maps: a neural circuit theory of autism,” Biological Psychiatry, Vol.42, pp.1138–1147, 1997. L. Gustafsson and A.P. Paplinski, “Self-organization of an artificial neural network subjected to attention shift impairments and familiarity preference, characteristics studied in autism,” J. Autism Dev. Disorders, Vol.34, No.2, pp.189–198, April 2004. L. Gustafsson and A.P. Paplinski, “An attempt in modeling autism using self-organizing maps,” Proc. Int. Conference on Neural Information Processing, pp.1815–1818, November 2002. A.P. Paplinski, L. Gustafsson, “An attempt in modeling early intervention in autism using neural networks,” Proc. of the IEEE Int. Joint Conf. on Neural Networks, July 2004. L.A.V. de Carvalho, N. de Carvalho Ferreira and A. Fiszman, "A neurocomputational model for autism," Proceedings of the IV Brazilian Conf. on neural networks, pp.344–349, July 1999. G. Noriega, “Self-organizing maps as a model of brain mechanisms potentially linked to autism,” IEEE Trans. Neural Syst. Rehabil. Eng., Vol.15, No.2, pp.217-226, 2007. I. Cohen, “An artificial neural network analogue of learning in autism,” Biological Psychiatry, Vol.36, pp.5–20, 1994. I. Cohen, “Neural network analysis of learning in autism,” in D.Stein & J. Ludick (Eds.), Neural networks and psychopathology, Cambridge, Cambridge University Press, pp.274–315, 1998. C. O'Loughlin and P. Thagard, “Autism and coherence: a computational model,” Mind and Language, 15, 4, pp.375–392, 2000. C. Balkenius and P. Bjorne, “First steps toward a computational theory of autism,” Proceedings of the Fourth Internat. Workshop on Epigenetic Robotics, August 2004. J.L. McClelland, “The basis for hyperspecificity in autism: a preliminary suggestion based on properties of neural nets,” J. Autism and Developmental Disorders, Vol.30, No.5, pp.497–502, 2000. J.D. Lewis, J.L. Elman, “Growth-related neural reorganization and the autism phenotype: a test of the hypothesis that altered brain growth leads to altered connectivity,” Dev. Science, Vol.11, No.1, pp.135–155, 2008. S. Grossberg and D. Seidman, “Neural dynamics of autistic behaviors: cognitive, emotional, and timing substrates,” Psychological Review, Vol.113, No.3, pp.483–525, July 2006. G. Noriega, “A neural model for compensation of sensory abnormalities in autism through feedback from a measure of global perception, ” IEEE Trans. Neural Netw., Vol.19, No.8, pp.1402–1414, 2008. Y.-H. Tan, C.-Y. Xi, S.-P. Jiang, B.-X. Shi, L.-B. Wang and L. Wang, “Auditory abnormalities in children with autism,” Open Journal of Psychiatry, Vol.2, pp.33–37, 2012. S.R. Leekam, C. Nieto, S.J. Libby, L. Wing and J. Gould, “Describing the sensory abnormalities of children with autism,” J. Autism and Dev. Disorders, Vol.37, pp.894–910, 2007.

1534-4320 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2014.2363775, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00150, Rev.1

[31] A. Ben-Sasson, S.A. Cermak, G.I. Orsmond, H. Tager-Flusberg, A.S. Carter, M.B. Kadlec and W. Dunn, “Extreme sensory modulation behaviors in toddlers with autism spectrum disorders,” The American Journal of Occupational Therapy, Vol.61, No.5, pp.584–592, 2007. [32] E.R. Kandel, J.H. Schwartz and T.M. Jessel (Eds.), Principles of neural science, 4th Edition, McGraw Hill, New York, Part III: Elementary Interactions Between Neurons: Synaptic Transmission, Part V: Perception, Part VIII: The Development of the Nervous System, 2000. [33] J.O. Pickles, An introduction to the physiology of hearing, 4th Edition, Emerald Group Pub., 2012. [34] S.E. Shore and S. Dehmel, “Somatosensory influences on brainstem and midbrain auditory nuclei,” in B.E. Stein (Ed.), The new handbook of multisensory processing, MIT Press, 2012. [35] B.E. Stein, T.R. Stanford, and B.A. Rowland, “The neural basis of multisensory integration in the midbrain: its organization and maturation,” Hearing Research, 258, 2009. [36] J.G. McHaffie, V. Fuentes-Santamaria, J.C. Alvarado, A.L. FuentesFarias, G. Gutierrez-Ospina, and B.E. Stein, “Anatomical features of the intrinsic circuitry underlying multisensory integration in the superior colliculus,” in B.E. Stein (Ed.), The new handbook of multisensory processing, MIT Press, 2012. [37] T.A. Hackett, “Multisensory convergence in the thalamus,” in B.E. Stein (Ed.), The new handbook of multisensory processing, MIT Press, 2012. [38] A. Falchier, C. Cappe, P. Barone, and C.E. Schroeder, “Sensory convergence in low-level cortices,” in B.E. Stein (Ed.), The new handbook of multisensory processing, MIT Press, 2012. [39] M.A. Meredith, K.J. Cios, A.R. McQuinston, H.K. Lim, L.P. Keniston, and H.R. Clemo “Neuroanatomical identification of multisensory convergence on higher-level cortical neurons,” in B.E. Stein (Ed.), The new handbook of multisensory processing, MIT Press, 2012. [40] N.R. Euliano and J.C. Principe, “A spatio-temporal memory based on SOMs with activity diffusion,” in E. Oja and S. Kaski (Eds.), Kohonen maps, Elsevier, 1999. [41] G. Noriega, “Modeling propagation delays in the development of SOMs – a parallel with abnormal brain growth in autism,” Neural Networks, Vol.21, No.2–3, pp.130–139, 2008. [42] M. Slaney, Auditory toolbox Version 2, Technical Report #1998-010, Interval Research Corporation, 1998. [43] T. Kohonen, Self-organizing maps, 2nd Ed., Springer-Verlag, 1997. [44] H. Ritter, T. Martinetz and K. Schulten, Neural computation and selforganizing maps – An introduction, Addison-Wesley Pub. Co., 1992. [45] J. Sirosh and R. Miikkulainen, “How lateral interaction develops in a self-organizing feature map,” Proc. IEEE Conference on Neural Networks, pp.1360–1365, San Francisco, April 1993. [46] C. Maurice (Ed.), G. Green and S.C. Luce (Co-Eds.), Behavioral intervention for young children with autism – a manual for parents and professionals, Pro-Ed, 1996. [47] T.J. Perrault Jr. and B.A. Rowland, “Fundamental properties underlying multisensory integration in the superior colliculus,” in B.E. Stein (Ed.), The new handbook of multisensory processing, MIT Press, 2012. [48] P. Lakatos, C. Kayser and C.E. Schroeder, “Multisensory processing of unisensory information in primary cortical areas,” in B.E. Stein (Ed.), The new handbook of multisensory processing, MIT Press, 2012. [49] D.C. Montgomery, Design and analysis of experiments, 3rd Edition, John Wiley & Sons, 1991.

11

Gerardo Noriega (M’88) received the B.S. degree in electronics and communications engineering from the Universidad Iberoamericana, Mexico City, Mexico, in 1983 and the M.Eng. degree in electrical engineering from the University of Toronto, Toronto, ON, Canada, in 1991. He is Vice-President and R&D Manager with RMS Instruments, Mississauga, ON, Canada, and has three decades of industrial experience in research and development work in the area of embedded real-time systems for mission-critical applications. His research interests include adaptive signal processing, estimation theory, neural modeling/engineering, and autism. He is sole or principal author of more than a dozen papers published in peer-reviewed journals and conference proceedings. Mr. Noriega is a registered Professional Engineer in the Province of Ontario.

1534-4320 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.