Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #8391

Re: NIO multiplexing + thread pooling

Show all headers | View raw

On 9/28/2011 9:56 AM, Robert Klemme wrote:

> It all depends of course how it's done. But imagine 10,000 open channels
> with traffic, 1,000 reader threads and a single thread doing selecting
> and dispatching to reader threads.


How much traffic?  One single 100 gigabit port running all out?

I think this scenario would depend on the number of packets received, 
not even actual traffic. 100 1 byte packets will require a lot more 
attention than 1 100 byte packet.  Packet reception should be where the 
bottleneck is.

I think you'd have to profile an actual application to determine where 
the time spend really is.  I'd like to see what someone with 10k open 
ports is actually doing, and what sort of hardware they're running on.


> If you do not partition (and this is
> the case I was talking about) you probably put something into a single
> queue from which those 1,000 reader threads read and the selector thread
> is hammering it all the time from the other end.


There's only one physical network controller.  Running flat out and with 
an average of 100 byte length packets (something like a typical HTTP GET 
request), I estimated that the network controller would have to write 
out one packet to memory every 30 nano seconds.

I'm not sure that's feasible.

Wikipedia indicates that DDR2 memory has a bandwidth of around 12 GB/s. 
  That's on the order of 100 Gb ethernet.  But "on the order of" means 
there's no bandwidth left for any other device, including the CPU or 
hard disc.

>
> Suggest a synchronization mechanism for the use case I described above
> (1 selector, 10,000 channels, 1,000 readers) and then we can talk about
> properties of that.


Show me an actual app first, I'm not sure what you've proposed is 
actually slightly realistic.

1 selector = 1 physical network device.  I'm not sure banging on one 
device with 10 threads is somehow better than just letting one thread 
interface with the device.

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

NIO multiplexing + thread pooling Giovanni Azua <bravegag@hotmail.com> - 2011-09-24 20:32 +0200
  Re: NIO multiplexing + thread pooling Tom Anderson <twic@urchin.earth.li> - 2011-09-24 20:46 +0100
    Re: NIO multiplexing + thread pooling Giovanni Azua <bravegag@hotmail.com> - 2011-09-24 23:09 +0200
    Re: NIO multiplexing + thread pooling Robert Klemme <shortcutter@googlemail.com> - 2011-09-25 11:33 +0200
      Re: NIO multiplexing + thread pooling Tom Anderson <twic@urchin.earth.li> - 2011-09-27 20:52 +0100
        Re: NIO multiplexing + thread pooling Robert Klemme <shortcutter@googlemail.com> - 2011-09-28 08:21 +0200
          Re: NIO multiplexing + thread pooling markspace <-@.> - 2011-09-28 08:20 -0700
            Re: NIO multiplexing + thread pooling Robert Klemme <shortcutter@googlemail.com> - 2011-09-28 18:56 +0200
              Re: NIO multiplexing + thread pooling markspace <-@.> - 2011-09-28 11:07 -0700
            Re: NIO multiplexing + thread pooling Peter Duniho <NpOeStPeAdM@NnOwSlPiAnMk.com> - 2011-09-28 17:39 -0700
              Re: NIO multiplexing + thread pooling markspace <-@.> - 2011-09-29 09:25 -0700
                Re: NIO multiplexing + thread pooling Lew <lewbloch@gmail.com> - 2011-09-30 07:46 -0700
                Re: NIO multiplexing + thread pooling markspace <-@.> - 2011-09-30 08:22 -0700
                Re: NIO multiplexing + thread pooling Lew <lewbloch@gmail.com> - 2011-09-30 08:30 -0700
                Re: NIO multiplexing + thread pooling markspace <-@.> - 2011-09-30 08:49 -0700
                Re: NIO multiplexing + thread pooling Robert Klemme <shortcutter@googlemail.com> - 2011-09-30 20:57 +0200
  Re: NIO multiplexing + thread pooling "John B. Matthews" <nospam@nospam.invalid> - 2011-09-25 00:26 -0400
    Re: NIO multiplexing + thread pooling Giovanni Azua <bravegag@hotmail.com> - 2011-09-25 11:13 +0200

csiph-web