Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch > #2111 > unrolled thread

Cuda Q1: Multi Stage/Multiple Kernels/Multiple Passes Invokation Performance Penalty ?

Back to article view | Back to comp.arch

Contents

  Cuda Q1: Multi Stage/Multiple Kernels/Multiple Passes Invokation Performance Penalty ? "Skybuck Flying" <Windows7IsOK@DreamPC2006.com> - 2011-06-09 00:27 +0200
    Re: Cuda Q1: Multi Stage/Multiple Kernels/Multiple Passes Invokation Performance Penalty ? "Skybuck Flying" <Windows7IsOK@DreamPC2006.com> - 2011-06-09 00:49 +0200
    Re: Cuda Q1: Multi Stage/Multiple Kernels/Multiple Passes Invokation Performance Penalty ? "Andy \"Krazy\" Glew" <andy@SPAM.comp-arch.net> - 2011-06-11 10:09 -0700
      Re: Cuda Q1: Multi Stage/Multiple Kernels/Multiple Passes Invokation Performance Penalty ? "Skybuck Flying" <Windows7IsOK@DreamPC2006.com> - 2011-06-11 23:25 +0200

#2111 — Cuda Q1: Multi Stage/Multiple Kernels/Multiple Passes Invokation Performance Penalty ?

Hello,

Cuda Question 1:

(Perhaps there are some chaps in here that know a thing or two about 
nvidia's gpu's and the way they operate ;))

(Cuda Experience level: noobie Cuda Coding stage: planning)

I have an idea which would probably require the invokation of multiple 
kernels, one (different/unique) kernel for each stage (which can also be 
thought of as a "pass").

However I wonder if there is a "kernel invocation performance penalty" ?

The original/first idea was to embed all code into a single kernel, however 
that might be bad because of "branch divergence".

The new idea is to analyze the data in stage 1 and "collect and group" all 
data into "branch groups" so that each "branch group" can be executed 
independantly by a "branch execution kernel" at stage 2.

So each "divergent" branch would get it's own kernel. Thanks to compute 
capability 2.1 all these different kernels could be executed in parallel, or 
otherwise kernel-serial, but at least the threads inside the block (warp?) 
would execute in parallel without the branch since it was eliminated at 
stage 1.

So let's assume for the sake of discussion that there are 10 different 
kernels (stages) which all have to be executed in serial because of compute 
1.0 limitation or so.

Also let's assume all data has been allocated and no further memory 
allocations have to be done, all data is inside the gpu's main memory system 
so no cpu-to-gpu transfers needed ?!?

My question is the following:

Suppose the following code is executed:

Kernel 1
Kernel 2
Kernel 3
Kernel 4
etc
Kernel 10

Is there a performanc penalty for invoking a kernel ?

If so how much of a performance penalty ? How many cycles do it take for a 
kernel to start executing ?

In other words:

If the "branch/thread serialization" of a warp/all threads in a block is 
faster than invoking a kernel then it (the new idea) might not be worth it 
and everything should simply be stuffed into a single kernel and hope for 
the best (get data/non-divergance lucky).

Bye,
Skybuck.

[toc] | [next] | [standalone]

#2114

Google found this posting on the nvidia forum:

http://forums.nvidia.com/index.php?showtopic=84221

Which also reference this document:

http://forums.nvidia.com/index.php?app=core&module=attach&section=attach&attach_id=10366

Which claims the overhead in 2008 was about 24 microseconds, which is quite 
a lot for a gpu which is supposed to be 1.6 ghz now-a-days.

So I wonder if this is still the case and if this posting is still relevant 
or if this have perhaps changed ?

Some other noteworthy comments on that forum:

1. Kernel invocation overhead can be different per operating system.

2. Multiple small kernels might execute faster than a single super big 
kernel.

Bye,
  Skybuck. 

[toc] | [prev] | [next] | [standalone]

#2148

On 6/8/2011 3:27 PM, Skybuck Flying wrote:

> My question is the following:
>
> Suppose the following code is executed:
>
> Kernel 1
> Kernel 2
> Kernel 3
> Kernel 4
> etc
> Kernel 10
>
> Is there a performanc penalty for invoking a kernel ?
>
> If so how much of a performance penalty ? How many cycles do it take for
> a kernel to start executing ?


You are at the leading edge of GPU capabilities.

Just a few years ago, all GPUs (at least, all Nvidia and AMD/ATI) GPUs 
could only execute one kernel on the GPU at a time.  So the kernels 
could not execute in parallel.

If you had one kernel that only occupied a single SIMD engine, all of 
the other SIMD engines went idle until that kernel finished.

If you had a kernel that occupied M*N+1 SIMD engines, where N is the 
number in your GPU ...   you should be able to see the  fragmentation.

Now, I must admit that I have trouble following the incremental steps 
here.  It seems obvious  to me that ultimately we want to have different 
kernels from different users running on different SIMDs at the same time.

But, as far as I can tell, the current state of the art is that

* Nvidia allows different kernels from the same task to run 
simultaneously.  But not different kernels from different tasks.

* AMD/ATI doesn't alllow this at all at the moment, but is rumoured to 
be about to leapfrog Nvidia's capability.


I expect that I will be corrected by people who know the details.


I conjecture that these limitations are in part due to limited memory 
protexction hardware in the GPUs.  GPUs do have TLBs and the like, but 
details are sparse: I conjecture that they cannot currently handle 
different user privileges at the same time, that the entire GPU is in 
only a single security context.



Similar problems held  supercomputers back for years.   I remember that 
the Transputer Meiko computing surface could not handle two different 
tasks at the same time.

[toc] | [prev] | [next] | [standalone]

#2154

CUDA 4.0 has stream objects which can encompass multiple kernels.

I wonder if streams can reduce "kernel invocation overhead".

Bye,
  Skybuck.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.arch

csiph-web