Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch > #3088

Re: IBM demos cognitive computer chips

Show all headers | View raw

On 8/20/2011 12:15 AM, BGB wrote:
> On 8/19/2011 12:50 PM, Robert Myers wrote:
>> On 8/19/2011 2:52 PM, BGB wrote:
>>
>>>
>>> a problem I think is the lack of much in-depth information as to what
>>> exactly neurons actually do, and to some extent, what is done by a lot
>>> of electronic neural nets sounds mostly like glorified vector/matrix
>>> multiplies combined with a feedback-driven adaptation mechanism or
>>> similar.
>>>
>>
>> You said a mouthful.
>>
>
> well, mostly just stating it how it comes off (from what things I have
> read, ...).
>
>
> if one has N axons, and N dendrites, with a 1:1 mapping, and the output
> is the sum of each axon multiplied by each weight, essentially this is a
> dot-product.

Yes, although in some aspects it is even simpler due to the threshold 
function (see below).  Also note that you are looking at the "clocked" 
or time static version, but the proposed device uses the asynchronous, 
or spiked version which uses both "spatial" (multiple axons) and 
"temporal" spikes accumulate over time with a decay function.


> doing this for N outputs, essentially we have an N input vector being
> transformed into an N output vector via N dot products each with N
> multiplies.
>
> essentially, this is a vector*matrix multiply.
>
> looking on Wikipedia: apparently this is about right, except that each
> output is also generally scaled by 1/N run through tanh (appears to
> serve mostly as a range clamp function).

Using Tanh specifically is not critical (the logistic function is often 
used), but the function is required for more than range clamping. 
Specifically, it must be a non-linear function.  Without that, you have 
a Perceptron which Minsky showed is quite limited.  The output of the 
non linear function is then put through a threshold function that 
determined whether the "neuron" fires or not.  Non firing "neurons" are 
what simplifies the dot product as their output is zero so they don't 
contribute to the activation of the next "neuron".

This combination of a non-linear function and a threshold are what makes 
the problem interesting.

The fact that the chip is using the asynchronous spike model may also be 
significant.  In this version, which is closer to how the real brain 
works (i.e. no central clock) each "neuron" processes its inputs 
asynchronously from the other neurons and sends not a "level" as output 
but a "spike".  The inputs to a neuron are a time series of spikes which 
are then weighted as you described to determine if the "neuron" should fire.


> then presumably this is followed by updating the weights, say, for each:
> relation[j][i]=axon[i]*output[j];
> f=weight[j][i]*(1.0-rate)+relation[j][i]*rate;
> weight[j][i]=clamp(f, -1.0, 1.0); //maybe tanh here as well?
> then possibly scaling weight[j] by 1.0/(sigma(N,i=0)weight[j][i])
> (above just a naive guess, I don't know what they are actually using).
>
> granted, there is probably a lot more to it than this though, but I
> haven't really looked much into it.

I haven't seen specifically what they are doing, though the article says 
they are following the Hebb rule (strengthen the weights of those 
connections that are active when the neuron fires and weaken the 
others).  But remember, an input firing is a binary function, not a 
linear one as your equations seem to imply.


>>>
>>> granted, I also dislike how much some people are prone to
>>> trivialize/demean the Von Neumann architecture, as it is well proven at,
>>> actually working, and being generally useful... (vs most of its "better"
>>> alternatives).
>>>
>>
>> Actually, I belong to the camp that thinks that the Von Neumann
>> architecture has been worked to the boundaries of its inherent
>> limitations, and well beyond, actually.
>>
>
> possibly, but the architecture is also well proven (it has worked well
> for a long time), and switching to a fundamentally different
> architecture (if it, for example, breaks software compatibility) would
> be a bit of a hard sell.
>
> granted, internally many chips are using the Harvard architecture, but
> externally look and behave as if they followed the Von Neumann
> architecture (for example, we traditionally have a big flat address
> space with both code and data pages).
>
> it is possible that whatever is done internally, will need to behave as
> if it followed this model for sake of being usable.

You seem to be assuming that neural net computers need to be 
"programmed", hence should allow programmers to do so in a manner they 
are used to.  However, one of their interesting characteristics is that 
they are not "programmed" per se.  They are "trained" in a variety of ways.


> however, it is unlikely neural-nets could effectively replace
> traditional architectures, if anything, because they would be ill-suited
> to running existing software.

Right.  Remember, traditional computers were developed to solve problems 
that a real neural network, our brains were not good at.  Hence they are 
quite different.


> so, if anything, they would likely have to coexist, with software spread
> between multiple devices.

Yup.  just like we use traditional computers in combination with out own 
biological "computers" to solve problems.

> more so, I have doubts that a "fully generic" neural net (every neuron
> potentially connected to every other) can likely be easily scaled up to
> the proposed levels (millions of neurons), without needing to compromise
> somewhere (essentially, creating some sort of "auxiliary architecture"
> within the network):
> say, some sets of neurons connect to adjacent neighbors;
> other ones connect between nearby clusters;
> others are linked across the network, or represent a shared "global
> signaling state" or similar.

Nature is ahead of you here.  Look up cerebral columns.


> ultimately, the "shape" of the above network could influence the
> behavior of the network as a whole.

Probably right.



-- 
  - Stephen Fuld
(e-mail address disguised to prevent spam)

Back to comp.arch | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

IBM demos cognitive computer chips Brett Davis <ggtgp@yahoo.com> - 2011-08-18 18:02 -0500
  Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-18 20:37 -0400
    Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-19 13:40 -0400
      Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-19 11:52 -0700
        Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-19 15:50 -0400
          Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-20 00:15 -0700
            Re: IBM demos cognitive computer chips Stephen Fuld <SFuld@alumni.cmu.edu.invalid> - 2011-08-20 09:22 -0700
              Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-20 14:10 -0400
        Re: IBM demos cognitive computer chips Quadibloc <jsavard@ecn.ab.ca> - 2011-08-21 05:16 -0700
          Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-21 11:56 -0700
            Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-21 15:29 -0400
              Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-21 13:19 -0700
            Re: IBM demos cognitive computer chips "Andy \"Krazy\" Glew" <andy@SPAM.comp-arch.net> - 2011-08-22 09:41 -0400
              Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-22 12:22 -0700
                Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-22 13:01 -0700
              Re: IBM demos cognitive computer chips "Andy \"Krazy\" Glew" <andy@SPAM.comp-arch.net> - 2011-08-23 10:11 -0400
              Re: IBM demos cognitive computer chips Eric  Northup <digitaleric@gmail.com> - 2011-08-23 09:36 -0700
                Re: IBM demos cognitive computer chips "Andy \"Krazy\" Glew" <andy@SPAM.comp-arch.net> - 2011-08-24 22:15 -0400
                Re: IBM demos cognitive computer chips "Andy \"Krazy\" Glew" <andy@SPAM.comp-arch.net> - 2011-08-25 09:07 -0400
      Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-20 01:41 -0400
      Re: IBM demos cognitive computer chips Stephen Fuld <SFuld@alumni.cmu.edu.invalid> - 2011-08-20 08:55 -0700
        Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-20 16:10 -0400
          Re: IBM demos cognitive computer chips Stephen Fuld <SFuld@alumni.cmu.edu.invalid> - 2011-08-21 21:11 -0700
            Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-22 12:04 -0400
              Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-22 12:22 -0400
            Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-22 14:24 -0400
              Re: IBM demos cognitive computer chips Quadibloc <jsavard@ecn.ab.ca> - 2011-08-22 19:32 -0700
                Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-22 23:21 -0700
                Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-23 09:49 -0700
                Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-23 09:46 -0700
                Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-23 15:05 -0700
                Re: IBM demos cognitive computer chips Piotr Wyderski <peter.pan@neverland.mil> - 2011-08-26 16:04 +0200
                Re: IBM demos cognitive computer chips Quadibloc <jsavard@ecn.ab.ca> - 2011-08-26 09:58 -0700
                Re: IBM demos cognitive computer chips ChrisQ <meru@devnull.com> - 2011-08-29 14:56 +0100
                Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-29 11:06 -0400
                Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-29 16:10 -0400
                Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-29 17:48 -0400
                Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-29 19:24 -0400
    Re: IBM demos cognitive computer chips ggtgp@yahoo.com - 2012-03-30 21:54 -0700

csiph-web