Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch > #3080 > unrolled thread

IBM demos cognitive computer chips

Back to article view | Back to comp.arch

Contents

  IBM demos cognitive computer chips Brett Davis <ggtgp@yahoo.com> - 2011-08-18 18:02 -0500
    Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-18 20:37 -0400
      Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-19 13:40 -0400
        Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-19 11:52 -0700
          Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-19 15:50 -0400
            Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-20 00:15 -0700
              Re: IBM demos cognitive computer chips Stephen Fuld <SFuld@alumni.cmu.edu.invalid> - 2011-08-20 09:22 -0700
                Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-20 14:10 -0400
          Re: IBM demos cognitive computer chips Quadibloc <jsavard@ecn.ab.ca> - 2011-08-21 05:16 -0700
            Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-21 11:56 -0700
              Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-21 15:29 -0400
                Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-21 13:19 -0700
              Re: IBM demos cognitive computer chips "Andy \"Krazy\" Glew" <andy@SPAM.comp-arch.net> - 2011-08-22 09:41 -0400
                Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-22 12:22 -0700
                  Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-22 13:01 -0700
                Re: IBM demos cognitive computer chips "Andy \"Krazy\" Glew" <andy@SPAM.comp-arch.net> - 2011-08-23 10:11 -0400
                Re: IBM demos cognitive computer chips Eric  Northup <digitaleric@gmail.com> - 2011-08-23 09:36 -0700
                  Re: IBM demos cognitive computer chips "Andy \"Krazy\" Glew" <andy@SPAM.comp-arch.net> - 2011-08-24 22:15 -0400
                    Re: IBM demos cognitive computer chips "Andy \"Krazy\" Glew" <andy@SPAM.comp-arch.net> - 2011-08-25 09:07 -0400
        Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-20 01:41 -0400
        Re: IBM demos cognitive computer chips Stephen Fuld <SFuld@alumni.cmu.edu.invalid> - 2011-08-20 08:55 -0700
          Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-20 16:10 -0400
            Re: IBM demos cognitive computer chips Stephen Fuld <SFuld@alumni.cmu.edu.invalid> - 2011-08-21 21:11 -0700
              Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-22 12:04 -0400
                Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-22 12:22 -0400
              Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-22 14:24 -0400
                Re: IBM demos cognitive computer chips Quadibloc <jsavard@ecn.ab.ca> - 2011-08-22 19:32 -0700
                  Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-22 23:21 -0700
                    Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-23 09:49 -0700
                  Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-23 09:46 -0700
                    Re: IBM demos cognitive computer chips BGB <cr88192@hotmail.com> - 2011-08-23 15:05 -0700
                    Re: IBM demos cognitive computer chips Piotr Wyderski <peter.pan@neverland.mil> - 2011-08-26 16:04 +0200
                    Re: IBM demos cognitive computer chips Quadibloc <jsavard@ecn.ab.ca> - 2011-08-26 09:58 -0700
                    Re: IBM demos cognitive computer chips ChrisQ <meru@devnull.com> - 2011-08-29 14:56 +0100
                      Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-29 11:06 -0400
                        Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-29 16:10 -0400
                          Re: IBM demos cognitive computer chips EricP <ThatWouldBeTelling@thevillage.com> - 2011-08-29 17:48 -0400
                            Re: IBM demos cognitive computer chips Robert Myers <rbmyersusa@gmail.com> - 2011-08-29 19:24 -0400
      Re: IBM demos cognitive computer chips ggtgp@yahoo.com - 2012-03-30 21:54 -0700

Page 1 of 2  [1] 2  Next page →

#3080 — IBM demos cognitive computer chips

IBM demos cognitive computer chips
http://www.eetimes.com/electronics-news/4218883/IBM-demos-cognitive-computer-chips

DARPA shells out $21m for IBM cat brain chip
http://www.theregister.co.uk/2011/08/18/ibm_darpa_synapse_project/

The direction toward a AI brain is clear.
We can build it, as I pointed out before.

[toc] | [next] | [standalone]

#3081

Brett Davis wrote:
> IBM demos cognitive computer chips
> http://www.eetimes.com/electronics-news/4218883/IBM-demos-cognitive-computer-chips
> 
> DARPA shells out $21m for IBM cat brain chip
> http://www.theregister.co.uk/2011/08/18/ibm_darpa_synapse_project/
> 
> The direction toward a AI brain is clear.
> We can build it, as I pointed out before.

Following the bread crumbs... also see

http://en.wikipedia.org/wiki/SyNAPSE

This is a good description:

Fuzzy logic and memristive hardware
http://www.neurdon.com/2011/08/09/development-of-massively-parallel-fuzzy-adaptive-neural-algorithms-for-efficient-implementation-on-memristive-hardware/

http://nl.bu.edu/research/overview/

Adaptive Computation with Memristive Memory (2011)
http://maxversace.com/files/IEEE_Computer_cover_feature_Feb_2011.pdf

Eric

[toc] | [prev] | [next] | [standalone]

#3082

On 8/18/2011 8:37 PM, EricP wrote:
> Brett Davis wrote:
>> IBM demos cognitive computer chips
>> http://www.eetimes.com/electronics-news/4218883/IBM-demos-cognitive-computer-chips
>>
>>
>> DARPA shells out $21m for IBM cat brain chip
>> http://www.theregister.co.uk/2011/08/18/ibm_darpa_synapse_project/
>>
>> The direction toward a AI brain is clear.
>> We can build it, as I pointed out before.
>
> Following the bread crumbs... also see
>
> http://en.wikipedia.org/wiki/SyNAPSE
>
> This is a good description:
>
> Fuzzy logic and memristive hardware
> http://www.neurdon.com/2011/08/09/development-of-massively-parallel-fuzzy-adaptive-neural-algorithms-for-efficient-implementation-on-memristive-hardware/
>
>
> http://nl.bu.edu/research/overview/
>
> Adaptive Computation with Memristive Memory (2011)
> http://maxversace.com/files/IEEE_Computer_cover_feature_Feb_2011.pdf
>

I was discouraged by the articles, to the extent that I could force 
myself to spend time with them, because they all seemed to jumble up 
three very distinct steps in technological development:

1. Conceptual understanding.  That is to say, the process of asking and 
answering the question: "Do we understand, even in concept, how such and 
such a thing is done or might be done?"

2. Modeling and hypothesis testing.  The first two steps can't be 
completely distinct.  We get an idea.  We test it.  It stinks.  We look 
for a new idea.

3. Design for real-world applications.  Can the method that we have 
discovered that actually works be achieved with hardware that can 
actually be built to satisfy some real world need?

DARPA and most engineers seem genetically programmed to get step three 
mixed up with steps 1 and 2.

The first question about anything that looks like what the man on the 
street would recognize as artificial intelligence is: do we have the 
faintest idea, even in concept, as to how to do the problem AT ALL?

As I currently understand the matter, the answer to that last question 
is that no, we do not have any idea at all of how nature builds or how 
anyone could build a device recognizably endowed with artificial 
intelligence.

To me, that means we are at steps one and two.  How could we possibly 
worry about the third step at this point?  Who cares if the brain fits 
it all into a shoebox and twenty watts?  We don't know how to do it 
given the entire electrical grid and all of the sovereign territory of 
the United States of America.

Thus, who cares if memristors might conceivably have some advantage in 
some device that could conceivably be built at some unknown time in the 
future, because we have no idea how such a device would work, anyway?

What we have here, ladies and gentlemen, is a neat idea desperately in 
search of an application that needs it.  I can save everyone a lot of 
time: artificial intelligence research does not currently need memristors.

Robert.

[toc] | [prev] | [next] | [standalone]

#3083

On 8/19/2011 10:40 AM, Robert Myers wrote:
> On 8/18/2011 8:37 PM, EricP wrote:
>> Brett Davis wrote:
>>> IBM demos cognitive computer chips
>>> http://www.eetimes.com/electronics-news/4218883/IBM-demos-cognitive-computer-chips
>>>
>>>
>>>
>>> DARPA shells out $21m for IBM cat brain chip
>>> http://www.theregister.co.uk/2011/08/18/ibm_darpa_synapse_project/
>>>
>>> The direction toward a AI brain is clear.
>>> We can build it, as I pointed out before.
>>
>> Following the bread crumbs... also see
>>
>> http://en.wikipedia.org/wiki/SyNAPSE
>>
>> This is a good description:
>>
>> Fuzzy logic and memristive hardware
>> http://www.neurdon.com/2011/08/09/development-of-massively-parallel-fuzzy-adaptive-neural-algorithms-for-efficient-implementation-on-memristive-hardware/
>>
>>
>>
>> http://nl.bu.edu/research/overview/
>>
>> Adaptive Computation with Memristive Memory (2011)
>> http://maxversace.com/files/IEEE_Computer_cover_feature_Feb_2011.pdf
>>
>
> I was discouraged by the articles, to the extent that I could force
> myself to spend time with them, because they all seemed to jumble up
> three very distinct steps in technological development:
>
> 1. Conceptual understanding. That is to say, the process of asking and
> answering the question: "Do we understand, even in concept, how such and
> such a thing is done or might be done?"
>
> 2. Modeling and hypothesis testing. The first two steps can't be
> completely distinct. We get an idea. We test it. It stinks. We look for
> a new idea.
>
> 3. Design for real-world applications. Can the method that we have
> discovered that actually works be achieved with hardware that can
> actually be built to satisfy some real world need?
>
> DARPA and most engineers seem genetically programmed to get step three
> mixed up with steps 1 and 2.
>
> The first question about anything that looks like what the man on the
> street would recognize as artificial intelligence is: do we have the
> faintest idea, even in concept, as to how to do the problem AT ALL?
>
> As I currently understand the matter, the answer to that last question
> is that no, we do not have any idea at all of how nature builds or how
> anyone could build a device recognizably endowed with artificial
> intelligence.
>
> To me, that means we are at steps one and two. How could we possibly
> worry about the third step at this point? Who cares if the brain fits it
> all into a shoebox and twenty watts? We don't know how to do it given
> the entire electrical grid and all of the sovereign territory of the
> United States of America.
>
> Thus, who cares if memristors might conceivably have some advantage in
> some device that could conceivably be built at some unknown time in the
> future, because we have no idea how such a device would work, anyway?
>
> What we have here, ladies and gentlemen, is a neat idea desperately in
> search of an application that needs it. I can save everyone a lot of
> time: artificial intelligence research does not currently need memristors.
>

it is possible...

I would just have imagined using something like a GPU for doing neural 
nets. in this case, one would mostly need a large number of cores and 
the ability to evaluate things reasonably quickly (an external 10Hz or 
100Hz or whatever "frame image" would be simulated mostly by having the 
cores go and update it, and for the next frame, it is recalculated again).

the rest would then be something analogous to shader programs and 
texture-maps (and probably having to make use of OpenCL or similar).

a problem I think is the lack of much in-depth information as to what 
exactly neurons actually do, and to some extent, what is done by a lot 
of electronic neural nets sounds mostly like glorified vector/matrix 
multiplies combined with a feedback-driven adaptation mechanism or similar.


granted, I also dislike how much some people are prone to 
trivialize/demean the Von Neumann architecture, as it is well proven at, 
actually working, and being generally useful... (vs most of its "better" 
alternatives).


or such...

[toc] | [prev] | [next] | [standalone]

#3084

On 8/19/2011 2:52 PM, BGB wrote:

>
> a problem I think is the lack of much in-depth information as to what
> exactly neurons actually do, and to some extent, what is done by a lot
> of electronic neural nets sounds mostly like glorified vector/matrix
> multiplies combined with a feedback-driven adaptation mechanism or similar.
>

You said a mouthful.

>
> granted, I also dislike how much some people are prone to
> trivialize/demean the Von Neumann architecture, as it is well proven at,
> actually working, and being generally useful... (vs most of its "better"
> alternatives).
>

Actually, I belong to the camp that thinks that the Von Neumann 
architecture has been worked to the boundaries of its inherent 
limitations, and well beyond, actually.

The proposed program sounds a bit like surveying the battlefield at 
Gettysburg on July 4, 1863 with a design for a new rifle and claiming 
that you could have changed the outcome with your technology (maybe a 
plausible claim if your new technology was a modern assault rifle).

As a way of exploring wildly different ways of doing computing, the IBM, 
DARPA, and BU programs might actually make sense.  As yet another attack 
on the problem of artificial intelligence, it makes about as much sense 
as Pickett's charge.

Robert.

[toc] | [prev] | [next] | [standalone]

#3086

On 8/19/2011 12:50 PM, Robert Myers wrote:
> On 8/19/2011 2:52 PM, BGB wrote:
>
>>
>> a problem I think is the lack of much in-depth information as to what
>> exactly neurons actually do, and to some extent, what is done by a lot
>> of electronic neural nets sounds mostly like glorified vector/matrix
>> multiplies combined with a feedback-driven adaptation mechanism or
>> similar.
>>
>
> You said a mouthful.
>

well, mostly just stating it how it comes off (from what things I have 
read, ...).


if one has N axons, and N dendrites, with a 1:1 mapping, and the output 
is the sum of each axon multiplied by each weight, essentially this is a 
dot-product.

doing this for N outputs, essentially we have an N input vector being 
transformed into an N output vector via N dot products each with N 
multiplies.

essentially, this is a vector*matrix multiply.

looking on Wikipedia: apparently this is about right, except that each 
output is also generally scaled by 1/N run through tanh (appears to 
serve mostly as a range clamp function).


then presumably this is followed by updating the weights, say, for each:
relation[j][i]=axon[i]*output[j];
f=weight[j][i]*(1.0-rate)+relation[j][i]*rate;
weight[j][i]=clamp(f, -1.0, 1.0); //maybe tanh here as well?
then possibly scaling weight[j] by 1.0/(sigma(N,i=0)weight[j][i])
(above just a naive guess, I don't know what they are actually using).

granted, there is probably a lot more to it than this though, but I 
haven't really looked much into it.



>>
>> granted, I also dislike how much some people are prone to
>> trivialize/demean the Von Neumann architecture, as it is well proven at,
>> actually working, and being generally useful... (vs most of its "better"
>> alternatives).
>>
>
> Actually, I belong to the camp that thinks that the Von Neumann
> architecture has been worked to the boundaries of its inherent
> limitations, and well beyond, actually.
>

possibly, but the architecture is also well proven (it has worked well 
for a long time), and switching to a fundamentally different 
architecture (if it, for example, breaks software compatibility) would 
be a bit of a hard sell.

granted, internally many chips are using the Harvard architecture, but 
externally look and behave as if they followed the Von Neumann 
architecture (for example, we traditionally have a big flat address 
space with both code and data pages).

it is possible that whatever is done internally, will need to behave as 
if it followed this model for sake of being usable.

an example would be if ones' neural-net chips showed up as large 
memory-mapped devices (where neural activation, weights, ... could be 
inspected and modified as needed, or saved/restored from disk files, ...).


however, it is unlikely neural-nets could effectively replace 
traditional architectures, if anything, because they would be ill-suited 
to running existing software.

so, if anything, they would likely have to coexist, with software spread 
between multiple devices.


more so, I have doubts that a "fully generic" neural net (every neuron 
potentially connected to every other) can likely be easily scaled up to 
the proposed levels (millions of neurons), without needing to compromise 
somewhere (essentially, creating some sort of "auxiliary architecture" 
within the network):
say, some sets of neurons connect to adjacent neighbors;
other ones connect between nearby clusters;
others are linked across the network, or represent a shared "global 
signaling state" or similar.

ultimately, the "shape" of the above network could influence the 
behavior of the network as a whole.


> The proposed program sounds a bit like surveying the battlefield at
> Gettysburg on July 4, 1863 with a design for a new rifle and claiming
> that you could have changed the outcome with your technology (maybe a
> plausible claim if your new technology was a modern assault rifle).
>
> As a way of exploring wildly different ways of doing computing, the IBM,
> DARPA, and BU programs might actually make sense. As yet another attack
> on the problem of artificial intelligence, it makes about as much sense
> as Pickett's charge.
>


fair enough.

I don't think it particularly likely that either neural net chips, or 
for that matter, conventional x86 or ARM chips, or GPUs, will suddenly 
lead to "intelligence".

actually, at the moment I more lean towards a "weak AI" position, 
whereby one can emulate some aspects of intelligent agents, but I have 
my doubts as to whether or not human-like intelligence (Strong AI) is 
necessarily actually possible (in the near term, or possibly at all).

[toc] | [prev] | [next] | [standalone]

#3088

On 8/20/2011 12:15 AM, BGB wrote:
> On 8/19/2011 12:50 PM, Robert Myers wrote:
>> On 8/19/2011 2:52 PM, BGB wrote:
>>
>>>
>>> a problem I think is the lack of much in-depth information as to what
>>> exactly neurons actually do, and to some extent, what is done by a lot
>>> of electronic neural nets sounds mostly like glorified vector/matrix
>>> multiplies combined with a feedback-driven adaptation mechanism or
>>> similar.
>>>
>>
>> You said a mouthful.
>>
>
> well, mostly just stating it how it comes off (from what things I have
> read, ...).
>
>
> if one has N axons, and N dendrites, with a 1:1 mapping, and the output
> is the sum of each axon multiplied by each weight, essentially this is a
> dot-product.

Yes, although in some aspects it is even simpler due to the threshold 
function (see below).  Also note that you are looking at the "clocked" 
or time static version, but the proposed device uses the asynchronous, 
or spiked version which uses both "spatial" (multiple axons) and 
"temporal" spikes accumulate over time with a decay function.


> doing this for N outputs, essentially we have an N input vector being
> transformed into an N output vector via N dot products each with N
> multiplies.
>
> essentially, this is a vector*matrix multiply.
>
> looking on Wikipedia: apparently this is about right, except that each
> output is also generally scaled by 1/N run through tanh (appears to
> serve mostly as a range clamp function).

Using Tanh specifically is not critical (the logistic function is often 
used), but the function is required for more than range clamping. 
Specifically, it must be a non-linear function.  Without that, you have 
a Perceptron which Minsky showed is quite limited.  The output of the 
non linear function is then put through a threshold function that 
determined whether the "neuron" fires or not.  Non firing "neurons" are 
what simplifies the dot product as their output is zero so they don't 
contribute to the activation of the next "neuron".

This combination of a non-linear function and a threshold are what makes 
the problem interesting.

The fact that the chip is using the asynchronous spike model may also be 
significant.  In this version, which is closer to how the real brain 
works (i.e. no central clock) each "neuron" processes its inputs 
asynchronously from the other neurons and sends not a "level" as output 
but a "spike".  The inputs to a neuron are a time series of spikes which 
are then weighted as you described to determine if the "neuron" should fire.


> then presumably this is followed by updating the weights, say, for each:
> relation[j][i]=axon[i]*output[j];
> f=weight[j][i]*(1.0-rate)+relation[j][i]*rate;
> weight[j][i]=clamp(f, -1.0, 1.0); //maybe tanh here as well?
> then possibly scaling weight[j] by 1.0/(sigma(N,i=0)weight[j][i])
> (above just a naive guess, I don't know what they are actually using).
>
> granted, there is probably a lot more to it than this though, but I
> haven't really looked much into it.

I haven't seen specifically what they are doing, though the article says 
they are following the Hebb rule (strengthen the weights of those 
connections that are active when the neuron fires and weaken the 
others).  But remember, an input firing is a binary function, not a 
linear one as your equations seem to imply.


>>>
>>> granted, I also dislike how much some people are prone to
>>> trivialize/demean the Von Neumann architecture, as it is well proven at,
>>> actually working, and being generally useful... (vs most of its "better"
>>> alternatives).
>>>
>>
>> Actually, I belong to the camp that thinks that the Von Neumann
>> architecture has been worked to the boundaries of its inherent
>> limitations, and well beyond, actually.
>>
>
> possibly, but the architecture is also well proven (it has worked well
> for a long time), and switching to a fundamentally different
> architecture (if it, for example, breaks software compatibility) would
> be a bit of a hard sell.
>
> granted, internally many chips are using the Harvard architecture, but
> externally look and behave as if they followed the Von Neumann
> architecture (for example, we traditionally have a big flat address
> space with both code and data pages).
>
> it is possible that whatever is done internally, will need to behave as
> if it followed this model for sake of being usable.

You seem to be assuming that neural net computers need to be 
"programmed", hence should allow programmers to do so in a manner they 
are used to.  However, one of their interesting characteristics is that 
they are not "programmed" per se.  They are "trained" in a variety of ways.


> however, it is unlikely neural-nets could effectively replace
> traditional architectures, if anything, because they would be ill-suited
> to running existing software.

Right.  Remember, traditional computers were developed to solve problems 
that a real neural network, our brains were not good at.  Hence they are 
quite different.


> so, if anything, they would likely have to coexist, with software spread
> between multiple devices.

Yup.  just like we use traditional computers in combination with out own 
biological "computers" to solve problems.

> more so, I have doubts that a "fully generic" neural net (every neuron
> potentially connected to every other) can likely be easily scaled up to
> the proposed levels (millions of neurons), without needing to compromise
> somewhere (essentially, creating some sort of "auxiliary architecture"
> within the network):
> say, some sets of neurons connect to adjacent neighbors;
> other ones connect between nearby clusters;
> others are linked across the network, or represent a shared "global
> signaling state" or similar.

Nature is ahead of you here.  Look up cerebral columns.


> ultimately, the "shape" of the above network could influence the
> behavior of the network as a whole.

Probably right.



-- 
  - Stephen Fuld
(e-mail address disguised to prevent spam)

[toc] | [prev] | [next] | [standalone]

#3089

Stephen Fuld wrote:
> On 8/20/2011 12:15 AM, BGB wrote:
>> On 8/19/2011 12:50 PM, Robert Myers wrote:
>>> On 8/19/2011 2:52 PM, BGB wrote:
>>>
>>>>
>>>> a problem I think is the lack of much in-depth information as to what
>>>> exactly neurons actually do, and to some extent, what is done by a lot
>>>> of electronic neural nets sounds mostly like glorified vector/matrix
>>>> multiplies combined with a feedback-driven adaptation mechanism or
>>>> similar.
>>>>
>>>
>>> You said a mouthful.
>>>
>>
>> well, mostly just stating it how it comes off (from what things I have
>> read, ...).
>>
>>
>> if one has N axons, and N dendrites, with a 1:1 mapping, and the output
>> is the sum of each axon multiplied by each weight, essentially this is a
>> dot-product.
> 
> Yes, although in some aspects it is even simpler due to the threshold 
> function (see below).  Also note that you are looking at the "clocked" 
> or time static version, but the proposed device uses the asynchronous, 
> or spiked version which uses both "spatial" (multiple axons) and 
> "temporal" spikes accumulate over time with a decay function.
> 
> 
>> doing this for N outputs, essentially we have an N input vector being
>> transformed into an N output vector via N dot products each with N
>> multiplies.
>>
>> essentially, this is a vector*matrix multiply.
>>
>> looking on Wikipedia: apparently this is about right, except that each
>> output is also generally scaled by 1/N run through tanh (appears to
>> serve mostly as a range clamp function).
> 
> Using Tanh specifically is not critical (the logistic function is often 
> used), but the function is required for more than range clamping. 
> Specifically, it must be a non-linear function.  Without that, you have 
> a Perceptron which Minsky showed is quite limited.  The output of the 
> non linear function is then put through a threshold function that 
> determined whether the "neuron" fires or not.  Non firing "neurons" are 
> what simplifies the dot product as their output is zero so they don't 
> contribute to the activation of the next "neuron".
> 
> This combination of a non-linear function and a threshold are what makes 
> the problem interesting.
> 
> The fact that the chip is using the asynchronous spike model may also be 
> significant.  In this version, which is closer to how the real brain 
> works (i.e. no central clock) each "neuron" processes its inputs 
> asynchronously from the other neurons and sends not a "level" as output 
> but a "spike".  The inputs to a neuron are a time series of spikes which 
> are then weighted as you described to determine if the "neuron" should 
> fire.
> 
> 
>> then presumably this is followed by updating the weights, say, for each:
>> relation[j][i]=axon[i]*output[j];
>> f=weight[j][i]*(1.0-rate)+relation[j][i]*rate;
>> weight[j][i]=clamp(f, -1.0, 1.0); //maybe tanh here as well?
>> then possibly scaling weight[j] by 1.0/(sigma(N,i=0)weight[j][i])
>> (above just a naive guess, I don't know what they are actually using).
>>
>> granted, there is probably a lot more to it than this though, but I
>> haven't really looked much into it.
> 
> I haven't seen specifically what they are doing, though the article says 
> they are following the Hebb rule (strengthen the weights of those 
> connections that are active when the neuron fires and weaken the 
> others).  But remember, an input firing is a binary function, not a 
> linear one as your equations seem to imply.

The same IBMers wrote these papers:

Anatomy of a Cortical Simulator (2007)
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.117.7473

The Cat is Out of the Bag:
Cortical Simulations with 10^9 Neurons, 10^13 Synapses (2009)
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.158.3759

"using Lawrence Livermore National Labs' state-of-the-art Dawn
Blue Gene/P (BG/P) supercomputer with 147,456 CPUs and 144 TB
of total memory, we achieve cortical simulations at an unprecedented
and historic scale exceeding that of cat cerebral cortex"

"We simulated a biologically-inspired model with 1.617*10^9 neurons
and 0.887*10^13 synapses, roughly 643 times slower than real-time"

so the latest project is to do the same thing in real-time without
a Blue Gene. Presumably because the necessity of a top-o-the-line
supercomputer limits the number of investigators in the field.

Unfortunately, the paper doesn't say what the "cat" thought.

Eric

[toc] | [prev] | [next] | [standalone]

#3093

On Aug 19, 12:52 pm, BGB <cr88...@hotmail.com> wrote:

> granted, I also dislike how much some people are prone to
> trivialize/demean the Von Neumann architecture,

This is simply a reaction to the "real world", which uses the von
Neumann architecture exclusively because it knows nothing better - and
is currently being dragged kicking and screaming to the use of
multiple cores because the chipmakers can't make one core faster any
more.

We have the technology to make more powerful computers than the ones
we have now, just not more powerful the way we want and can easily
use. So exploring the alternatives makes sense.

John Savard

[toc] | [prev] | [next] | [standalone]

#3094

On 8/21/2011 5:16 AM, Quadibloc wrote:
> On Aug 19, 12:52 pm, BGB<cr88...@hotmail.com>  wrote:
>
>> granted, I also dislike how much some people are prone to
>> trivialize/demean the Von Neumann architecture,
>
> This is simply a reaction to the "real world", which uses the von
> Neumann architecture exclusively because it knows nothing better - and
> is currently being dragged kicking and screaming to the use of
> multiple cores because the chipmakers can't make one core faster any
> more.
>

AMD Fusion is cool, although too bad the GPU's don't run something more 
x86-like, as this could allow interesting possibilities. it may take 
several generations though before the technology gets "more cool".


although, it is potentially a reasonable tradeoff, as potentially if it 
were just x86, then compiler writers would likely stick to conventions 
which as-is make threading rather costly on normal systems:
using a single large dedicated per-thread stack.

the use of individual/independent frames would allow more economical use 
of memory and address space.

granted, people could also argue "well, we have 64 bits now, why not 
just give each thread its own 4MB or 8MB (or more) stack? (CPS/... is 
for wimps...)".


> We have the technology to make more powerful computers than the ones
> we have now, just not more powerful the way we want and can easily
> use. So exploring the alternatives makes sense.
>

or, maybe it could be more like:
"in the future, there will be CPUs and OS's beyond peoples' current 
dreams, and they will be: x86 and Windows".


it is like, I read some about memristors, and was left thinking there is 
little reason they couldn't be used for building more chips for more 
traditional architectures.

although, they support non-volatile operation, which has some other 
interesting properties:
say, if RAM and the CPU could be entirely powered down, but still retain 
their state, allowing a "sleep mode" which uses almost no power, ...

some other possibilities came to mind, but it is unclear why they can't 
be done already with existing technology:
say, one can "freeze dry" an entire process into a disk file, possibly 
reboot the main OS or kernel, and then restore the process as-it-was.

an example of how this could be used would be an "in-place" OS reboot, 
where the entire underlying OS reboots, but the all application-level 
state remains the same.


or such...

[toc] | [prev] | [next] | [standalone]

#3095

On 8/21/2011 2:56 PM, BGB wrote:
> On 8/21/2011 5:16 AM, Quadibloc wrote:
>> On Aug 19, 12:52 pm, BGB<cr88...@hotmail.com> wrote:
>>
>>> granted, I also dislike how much some people are prone to
>>> trivialize/demean the Von Neumann architecture,
>>
>> This is simply a reaction to the "real world", which uses the von
>> Neumann architecture exclusively because it knows nothing better - and
>> is currently being dragged kicking and screaming to the use of
>> multiple cores because the chipmakers can't make one core faster any
>> more.
>>
>
> AMD Fusion is cool, although too bad the GPU's don't run something more
> x86-like, as this could allow interesting possibilities. it may take
> several generations though before the technology gets "more cool".
>

GPU's don't ameliorate the inherent limitations of the Von Neumann 
architecture.  To the contrary, they make the usual problems associated 
with moving data around even more painfully obvious.  Except in those 
cases where the entire working set can be fit into super-fast graphics 
memory, GPU's will be memory-bound (latency, bandwidth, or both), just 
like most realistic problems on any other Von Neumann architecture.

Robert.

[toc] | [prev] | [next] | [standalone]

#3098

On 8/21/2011 12:29 PM, Robert Myers wrote:
> On 8/21/2011 2:56 PM, BGB wrote:
>> On 8/21/2011 5:16 AM, Quadibloc wrote:
>>> On Aug 19, 12:52 pm, BGB<cr88...@hotmail.com> wrote:
>>>
>>>> granted, I also dislike how much some people are prone to
>>>> trivialize/demean the Von Neumann architecture,
>>>
>>> This is simply a reaction to the "real world", which uses the von
>>> Neumann architecture exclusively because it knows nothing better - and
>>> is currently being dragged kicking and screaming to the use of
>>> multiple cores because the chipmakers can't make one core faster any
>>> more.
>>>
>>
>> AMD Fusion is cool, although too bad the GPU's don't run something more
>> x86-like, as this could allow interesting possibilities. it may take
>> several generations though before the technology gets "more cool".
>>
>
> GPU's don't ameliorate the inherent limitations of the Von Neumann
> architecture. To the contrary, they make the usual problems associated
> with moving data around even more painfully obvious. Except in those
> cases where the entire working set can be fit into super-fast graphics
> memory, GPU's will be memory-bound (latency, bandwidth, or both), just
> like most realistic problems on any other Von Neumann architecture.
>

I think the AMD Fusion idea is that much of the time, the working set 
can be largely held in cache (nevermind if it is measured in kB).

another possibly nifty idea of memristors in a conventional 
architecture: much larger CPU cache.

say, if in the future, one can have 64MB, 256MB, or 1GB of CPU cache, 
then the lower bandwidth of external memory will be less of an issue.


it is much like having lots of RAM mostly covers for the relatively slow 
speed of the disk drives...

yeah... powering through life at 5400 RPM speeds...


misc:

went off and read about split stacks (AKA: segmented stacks), which are 
possibly a more practical idea (cheaper/...) than would be expecting 
native compilers to generally migrate over to CPS and/or allocate 
individual call-frames.

a slight uncertainty though would be if I were to try using similar on 
Windows (in my own VM) if there is a risk that this could leave the OS 
hopelessly confused (my current system uses a mix of the native ABI and 
heap-allocated call-frames, generally with the native stacks shared as a 
temporary-use scratchpad between any number of VM-managed "green threads").

but, a partial advantage is that, it can be supported by the ABI but 
need not be used by the underlying OS (the stack-check operation being 
essentially no-op).

a slight downside though is that split-stacks still don't (inherently) 
allow any good way to implement continuations. actually, a common way of 
implementing continuations, stack-copying, would actually be made 
somewhat worse with this strategy. would likely still require heap-based 
storage for lexical variable capture (needed by closures).


or such...

[toc] | [prev] | [next] | [standalone]

#3103

On 8/21/2011 2:56 PM, BGB wrote:

> AMD Fusion is cool, although too bad the GPU's don't run something more
> x86-like, as this could allow interesting possibilities. it may take
> several generations though before the technology gets "more cool".

There is little reason why the GPU/SIMD/SIMT/coherent threading hardware 
could not be running any reasonable instruction set.  Possibly including 
x86 or ARM.

I hesitate to say "GPU", since if you really wanted graphics you would 
have to extend the ISA quite a bit.  But if all you wanted was GPGPU, 
almost any ISA would do.

Note that the basic effect f GPU-style SIMD/SIMT/Coherent threading is 
to amortize instruction decode over more compute.  You still may not 
want to waste area on x86 decoding, but the net lossage is reduced. 
Also, GPU-stye encourages non-superscalar decoding, and decoding x86 one 
instruction at a time is much better than sperscalar x86 decode.

If I had known about coherent threading when I was at AMD, I would have 
proposed that the multi-cluster multi-threading that seems to be in 
Bulldozer be "opportunistically coherent", running GPU-style 
SIMD/SIMT/coherent instruction fetch when the threads were executing the 
same loop.

However: it is not at all clear to me that x86 has much advantage any 
longer.  Of course, Intel and to a lesser extent AMD want you to believe 
that x86 still matters, and it probably does for laptops and servers, 
but x86 probably would not matter mauch for cell phones - ARM 
compatibility matters more there - and there may be a brief window where 
there is an opportunity for non-x86 non-ARM ISAs to break through.





> although, it is potentially a reasonable tradeoff, as potentially if it
> were just x86, then compiler writers would likely stick to conventions
> which as-is make threading rather costly on normal systems:
> using a single large dedicated per-thread stack.
>
> the use of individual/independent frames would allow more economical use
> of memory and address space.
>
> granted, people could also argue "well, we have 64 bits now, why not
> just give each thread its own 4MB or 8MB (or more) stack? (CPS/... is
> for wimps...)".

This is one of the keys:

Interleaved stacks for when operating in lockste, or nearly so.

Uninterleaved stacks for MIMD parallel.

[toc] | [prev] | [next] | [standalone]

#3108

On 8/22/2011 6:41 AM, Andy "Krazy" Glew wrote:
> On 8/21/2011 2:56 PM, BGB wrote:
>
>> AMD Fusion is cool, although too bad the GPU's don't run something more
>> x86-like, as this could allow interesting possibilities. it may take
>> several generations though before the technology gets "more cool".
>
> There is little reason why the GPU/SIMD/SIMT/coherent threading hardware
> could not be running any reasonable instruction set. Possibly including
> x86 or ARM.
>
> I hesitate to say "GPU", since if you really wanted graphics you would
> have to extend the ISA quite a bit. But if all you wanted was GPGPU,
> almost any ISA would do.
>

I read before that I guess the current generation of Fusion chips run 
code in their own address space, and use VLIW. the next generation will 
have access to a shared address space (via an MMU), and use a 
differently designed special-purpose ISA (not sure, I am getting "memory 
flashes" of something like variable-length instructions in 32-bit units 
or similar with 7/8 bit register fields or similar).

don't have the PDF on hand at the moment it seems.


> Note that the basic effect f GPU-style SIMD/SIMT/Coherent threading is
> to amortize instruction decode over more compute. You still may not want
> to waste area on x86 decoding, but the net lossage is reduced. Also,
> GPU-stye encourages non-superscalar decoding, and decoding x86 one
> instruction at a time is much better than sperscalar x86 decode.
>

yeah.

I am not sure how instruction decoding works in real hardware.

before I wrote an x86 interpreter (in software) which used an 
instruction decoder based on pattern matching (it was based on a tweaked 
form of my disassembler).

I guess HW would probably work based on bit-patterns or similar (there 
seem to be lots of bit patterns in these instructions).


> If I had known about coherent threading when I was at AMD, I would have
> proposed that the multi-cluster multi-threading that seems to be in
> Bulldozer be "opportunistically coherent", running GPU-style
> SIMD/SIMT/coherent instruction fetch when the threads were executing the
> same loop.
>

yep. a drawback though is that it requires the code to be executing the 
same logic at the same time to use effectively, whereas independent 
threads can do independent things, although sadly one can't likely have 
nearly so many of them.


> However: it is not at all clear to me that x86 has much advantage any
> longer. Of course, Intel and to a lesser extent AMD want you to believe
> that x86 still matters, and it probably does for laptops and servers,
> but x86 probably would not matter mauch for cell phones - ARM
> compatibility matters more there - and there may be a brief window where
> there is an opportunity for non-x86 non-ARM ISAs to break through.
>

I think the major advantage that x86 has is related to backwards 
compatibility and pretty much any software including an x86-specific 
compiler/JIT and/or a lot of ASM code (note: ASM is just not for 
performance: there are many nifty things which can only really be done 
effectively in ASM).

otherwise, for example, one has to port a lot of this to a different 
ISA, such as ARM.


also, in any case, having all of the threaded/parallel vector units 
running the same ISA and in the same processes as the main programs 
greatly simplifies the task of running code on them.

sadly though, for GPU-like uses, yeah, one probably would need to modify 
the ISA some, possibly with both some extensions and also some removed 
features.


>
>
>> although, it is potentially a reasonable tradeoff, as potentially if it
>> were just x86, then compiler writers would likely stick to conventions
>> which as-is make threading rather costly on normal systems:
>> using a single large dedicated per-thread stack.
>>
>> the use of individual/independent frames would allow more economical use
>> of memory and address space.
>>
>> granted, people could also argue "well, we have 64 bits now, why not
>> just give each thread its own 4MB or 8MB (or more) stack? (CPS/... is
>> for wimps...)".
>
> This is one of the keys:
>
> Interleaved stacks for when operating in lockste, or nearly so.
>
> Uninterleaved stacks for MIMD parallel.
>

as well as possibly split-stacks.

I was originally imagining the possibility of independent ("allocated") 
call frames and CPS, but split-stacks could allow higher performance on 
existing HW.


for example, on x86, one would either need the ability to have special 
atomic instructions for basic linked-list operations, or would need 
something like a "cli/sti pair that does bus locking".

say:
lock_cli
... linked list operation ...
lock_sti

otherwise, one needs (slightly more costly) spinlocks or mutexes, which 
could add costs if used in an ABI.

the special linked-list ops could be something like:
movlkn reg, [mem]	;mov, link-next (gets first item, move to next)
movlkp [mem], reg	;mov, link-prev (add reg before first item)

where movlkn could do an operation like:
lea _temp0, [...]
mov rax, [_temp0]
mov _temp1, [rax]
mov [_temp0], _temp1

and movlkp like:
lea _temp0, [...]
mov _temp1, [_temp0]
mov [rax], _temp1
mov [_temp0], rax

all as atomic operations...

actually, another possible variation:
movlkn reg, [mem], imm
movlkp [mem], reg, imm
movlknz reg, [mem], imm	;like movlkn, but is "null-safe"

where imm adds an offset to be used for the link-field (allowing more 
freedom in linked-list layout).


somehow I imagine that bus-locking cli/sti forms might be cheaper (but 
also potentially evil as well, say if abused as some sort of generic 
"critical section" operation).


absent something like the above ideas, using individually allocated 
stack frames would likely be fairly expensive (vs, say, using 
split-stacks or similar).

granted, atomic linked-list operations could also be useful in 
implementing things like message queues, ... as well.

or such...

[toc] | [prev] | [next] | [standalone]

#3109

On 8/22/2011 12:22 PM, BGB wrote:
> On 8/22/2011 6:41 AM, Andy "Krazy" Glew wrote:
>> On 8/21/2011 2:56 PM, BGB wrote:
>>

<snip>

>>> although, it is potentially a reasonable tradeoff, as potentially if it
>>> were just x86, then compiler writers would likely stick to conventions
>>> which as-is make threading rather costly on normal systems:
>>> using a single large dedicated per-thread stack.
>>>
>>> the use of individual/independent frames would allow more economical use
>>> of memory and address space.
>>>
>>> granted, people could also argue "well, we have 64 bits now, why not
>>> just give each thread its own 4MB or 8MB (or more) stack? (CPS/... is
>>> for wimps...)".
>>
>> This is one of the keys:
>>
>> Interleaved stacks for when operating in lockste, or nearly so.
>>
>> Uninterleaved stacks for MIMD parallel.
>>
>
> as well as possibly split-stacks.
>
> I was originally imagining the possibility of independent ("allocated")
> call frames and CPS, but split-stacks could allow higher performance on
> existing HW.
>

<snip>

>
> absent something like the above ideas, using individually allocated
> stack frames would likely be fairly expensive (vs, say, using
> split-stacks or similar).
>
> granted, atomic linked-list operations could also be useful in
> implementing things like message queues, ... as well.
>


just realized that I had overlooked a cheaper possibility for allocated 
stack-frames in a multi-threaded system:
the frame allocator first tries to allocate from TLS (Thread Local 
Storage), and only falls back to a synchronized locking mechanism in the 
case where the TLS var is empty.

say:
.get_frame:
mov rax, [gs:...]	;get TLS var from OS
and rax, rax
jnz .got_one
;malloc more stack frames (implicitly synchronizes)
jmp .get_frame
.got_one:
;adjust link to next
mov rcx, [rax]
mov [gs:...], rcx
ret ;or similar (or possibly a return via a link register)
jmp r14	;return via register


lea r14, [.L0]
jmp __alloc_frame_N
.L0:
...


the advantage here is it utilizes the fact that typically threads have 
fairly predictable patterns.

possibly, a TLS var would be needed for each size of frame supported 
(this would be better with explicit OS support).


nevermind the bigger issue of existing ABIs, ...
again, split stacks still have an advantage in that they are at least 
slightly more ABI friendly (whereas general-purpose allocated 
call-frames would likely require essentially redesigning the ABI from 
the ground up, and would pose more problems for operating within an 
existing OS's, where OS's/... generally tend to assume things like the 
existence of a stack... hence a stack-based ABI would be problematic).

IIRC, this should not be an issue for OS interrupts, ... since although 
the CPU assumes a stack, each ring will use its own stack (the potential 
lack of a valid rSP in userspace should thus not effect this).


granted, synchronized linked-list ops or similar could still be useful, 
but are not likely critical.


or such...

[toc] | [prev] | [next] | [standalone]

#3113

On 8/22/2011 9:41 AM, Andy "Krazy" Glew wrote:
> On 8/21/2011 2:56 PM, BGB wrote:
>
>> AMD Fusion is cool, although too bad the GPU's don't run something more
>> x86-like, as this could allow interesting possibilities. it may take
>> several generations though before the technology gets "more cool".
>
> There is little reason why the GPU/SIMD/SIMT/coherent threading hardware
> could not be running any reasonable instruction set. Possibly including
> x86 or ARM.

I posted too soon: there is one reason.

The present generation of GPUs with SIMD/SIMT/coherent threading rely on 
annotations or indications in the instruction stream wrt control flow: 
IF-THEN-ELSEIF-ELSE-ENDIF, BEGIN/END-LOOP, etc.

This directs the controller to push/pop/exchange IPs and masks of 
selected processors on the single, interleaved, stack.

To do this coherent threading on a vanilla x86 or ARM ISA would require 
fairly complex mechanisms to deduce where these annotations should be. 
I know this from experience: these are exactly the sort of annotations 
that you need to do control independence optimizations, such as Haitham 
Akkary's DMT with fork loop bodies or fork after loop.

But adding these annotations would not be so hard to x86, at least, 
which already has extension bytes out the wazoo.   I am not so sure 
about ARM, but I note that Fujitsu added extensions (not bytes, but 
whole instruction word extensions) to SPARC.

[toc] | [prev] | [next] | [standalone]

#3115

On Aug 22, 6:41 am, "Andy \"Krazy\" Glew" <a...@SPAM.comp-arch.net>
wrote:
> On 8/21/2011 2:56 PM, BGB wrote:
> > the use of individual/independent frames would allow more economical use
> > of memory and address space.
>
> > granted, people could also argue "well, we have 64 bits now, why not
> > just give each thread its own 4MB or 8MB (or more) stack? (CPS/... is
> > for wimps...)".
>
> This is one of the keys:
>
> Interleaved stacks for when operating in lockste, or nearly so.
>
> Uninterleaved stacks for MIMD parallel.

I presume the benefit is for improved locality of memory references.

What would the granularity of interleaving be?  Does the instruction
stream know that it's stack has been interleaved, or do the stack
pointer arithmetic operations do the swizzling?

[toc] | [prev] | [next] | [standalone]

#3130

On 8/23/2011 12:36 PM, Eric Northup wrote:
> On Aug 22, 6:41 am, "Andy \"Krazy\" Glew"<a...@SPAM.comp-arch.net>
> wrote:
>> On 8/21/2011 2:56 PM, BGB wrote:
>>> the use of individual/independent frames would allow more economical use
>>> of memory and address space.
>>
>>> granted, people could also argue "well, we have 64 bits now, why not
>>> just give each thread its own 4MB or 8MB (or more) stack? (CPS/... is
>>> for wimps...)".
>>
>> This is one of the keys:
>>
>> Interleaved stacks for when operating in lockste, or nearly so.
>>
>> Uninterleaved stacks for MIMD parallel.
>
> I presume the benefit is for improved locality of memory references.
>
> What would the granularity of interleaving be?  Does the instruction
> stream know that it's stack has been interleaved, or do the stack
> pointer arithmetic operations do the swizzling?

Stack interleaving arises for reasons that include:

* when you have several coherent threads, some variables, such as loop 
counter variables, are common to all threads, while others are 
per-thread.  It is wasteful to push N copies of the same loop counter on 
separate stacks.

* but mainly (I think) because of the mindset where you think of the 
threads as not being independent, but instead being lockstep, with a 
single instruction pointer, and a mask indicating which threads are 
active and which are disabled.

e.g. an IF statement computes a mask of which threads take the THEN 
path, and pushes a mask for the threads that take the ELSE path, along 
with the PC.

an ELSE statement swaps the current mask and PC with the mask and PC at 
the top of the stack.

an ENDIF statement merges the current mask with the mask popped off the 
stack.



You can equally well think as these operations as involving separate PCs 
with separate stacks - but that does a lot of unnecessary work.  Plus, 
you would need to compare PCs popped off to verifypical y that you are 
still coherent.  Whereas coherence happens by construction with the 
"SIMD" PC and mask.




AFAIK present GPU instruction sets make the interleaving explicit. 
Indeed, some have a hardware stack in registers, adjunct to memory. 
(Note that you only need use the PC/mask stack on divergent branches, 
not every branch if all coherent threads go the same way, no need.)



But notice the limitations of this approach:

The interleaved stack only applies to the SIMD warp, not all of the SIMD 
groups.

Typical warps or wavefronts are 64 threads wide. (Or, rather, 16 wide 
times 4 cycles deep.)

There may be variables that are not common to all 64 threads in a group, 
but which are common to, say, every 4, or 16, or 4+16+3+1+40 arbitrary 
partitions.

Interleaved stacks tend to benefot only shared across all thread 
variables, but not partial sharing.

I would like to see studies on the relative benefit of this versus the 
lossage of not being able to go MIMD when you need to.

[toc] | [prev] | [next] | [standalone]

#3137

[Attributions - not misattributed, but not all quotees may have quotes 
included after editing]

On 8/24/2011 10:15 PM, Andy "Krazy" Glew wrote:
> On 8/23/2011 12:36 PM, Eric Northup wrote:
>> On Aug 22, 6:41 am, "Andy \"Krazy\" Glew"<a...@SPAM.comp-arch.net>
>> wrote:
>>> On 8/21/2011 2:56 PM, BGB wrote:


>>> Interleaved stacks for when operating in lockste, or nearly so.
>>>
>>> Uninterleaved stacks for MIMD parallel.
>>
>> I presume the benefit is for improved locality of memory references.
>...
> Stack interleaving arises for reasons that include:
>
> * when you have several coherent threads, some variables, such as loop
> counter variables, are common to all threads, while others are
> per-thread. It is wasteful to push N copies of the same loop counter on
> separate stacks.
>
> * but mainly (I think) because of the mindset where you think of the
> threads as not being independent, but instead being lockstep, with a
> single instruction pointer, and a mask indicating which threads are
> active and which are disabled.

It should be pointed out that "interleaved stack" is generic, and does 
not necessarily mean that there is a single stack for all work.  As 
Bernd Paysan points out (http://bernd-paysan.de/4stack.html), one stack 
is common, dual stack is forthright, and four stacks may bew what you 
want.  (OK, bad attempt at humor. :-)

The SIMD/SIMT/coherent threading control flow stack on which mask and 
PCs are pushed could be a microarchitecture stack entirely separate from 
all of the other stacks.  Of depth equal to the maximum number of 
threads in a workgroup, warp, or wavefront.

But if it is microarchitecture and you have a conventional stacks for 
the vector lane threads, then you need logic to verify, on occasion, 
whether the threads are still coherent.

A separate data stack that allows you to take advantage of cross-thread 
variables by pushing them only once, whereas you push a vector for 
per-thread variables, is straightforward - but may lead to misalignment 
if the allocation order on the stack is conventional.  This is exactly 
the problem that you have with vector machines - allocating scalars and 
vectors on the same stack leads to misalignment and/or padding. 
Separate stacks for scalars and vectors, or other sizes, may be desirable.

Finally, yes, as Eric pointed out, there is probably better locality for 
coherent threads with interleaved stacks.  Or, at least, fewer cache 
lines required.

I suspect that addressing these issues is one motivation of AMD 
apparently abandoning SMT/coherent threading in their next generation 
GPU, which I like to call "Larrabee done right".  The explicit vector 
ISA gives software more control over these data alignment issues.

Nevertheless, I suspect that somebody, perhaps not AMD, will come up 
with instructions that make it easier to do SIMT/instruction coherence 
using vector masks.

[toc] | [prev] | [next] | [standalone]

#3085

Robert Myers wrote:
> I was discouraged by the articles, to the extent that I could force 
> myself to spend time with them, because they all seemed to jumble up 
> three very distinct steps in technological development:
> 
> 1. Conceptual understanding.  That is to say, the process of asking and 
> answering the question: "Do we understand, even in concept, how such and 
> such a thing is done or might be done?"
> 
> 2. Modeling and hypothesis testing.  The first two steps can't be 
> completely distinct.  We get an idea.  We test it.  It stinks.  We look 
> for a new idea.
> 
> 3. Design for real-world applications.  Can the method that we have 
> discovered that actually works be achieved with hardware that can 
> actually be built to satisfy some real world need?
> 
> DARPA and most engineers seem genetically programmed to get step three 
> mixed up with steps 1 and 2.
> 
> The first question about anything that looks like what the man on the 
> street would recognize as artificial intelligence is: do we have the 
> faintest idea, even in concept, as to how to do the problem AT ALL?
> 
> As I currently understand the matter, the answer to that last question 
> is that no, we do not have any idea at all of how nature builds or how 
> anyone could build a device recognizably endowed with artificial 
> intelligence.
> 
> To me, that means we are at steps one and two.  How could we possibly 
> worry about the third step at this point?  Who cares if the brain fits 
> it all into a shoebox and twenty watts?  We don't know how to do it 
> given the entire electrical grid and all of the sovereign territory of 
> the United States of America.
> 
> Thus, who cares if memristors might conceivably have some advantage in 
> some device that could conceivably be built at some unknown time in the 
> future, because we have no idea how such a device would work, anyway?
> 
> What we have here, ladies and gentlemen, is a neat idea desperately in 
> search of an application that needs it.  I can save everyone a lot of 
> time: artificial intelligence research does not currently need memristors.
> 
> Robert.
> 

I think they want to create a tool that allows them to
experiment at levels they never had before and see what happens.
Ya know... emergent behavior and all that.
They seem to be particularly interested in image processing.
The current technology has
- 10's to 100's of neurons
- 2 to 5 connections/neuron
- 2D connections
- 2 or 3 layers

They want to build
- millions of neurons
- 10,000 connections/neuron
- 3D connections
- arbitrary layers

To do that, particularly 3D connections at those scales,
implies close packing, which is why they are shooting for
power density's of ~100 milliwatts/cm^2.

All this only became possible to even contemplate circa 2008 or 2009.
The key seems to be (a) that memristors can be used to mimic neurons
and (b) hybrids in CMOS allowing memristors + processors in memory.

Eric

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.arch

csiph-web