Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #9998

Re: JMS scalability question

Show all headers | View raw

Hi Fredrik,

> Your problem description is too generic to say anything about your
> performance. I'd claim a few thousands consumers shouldn't be a problem for
> a JMS broker worth its salt.

yes, my first post was not that specific in order to keep the problem
description short. If you have a look at my reply with the MyActor
class, there is much more information provided.

> On the other hand if those consumers are on the same jvm you'll now have
> thousands of threads fighting over local resourses.

For non-distributed communication between actors that all reside
inside the same vm I won't go with JMS, but with plain JDK1.5
BlockingQueues. What I want to develop is some kind of runtime
environment for actors to live in. A lot of the considerations you
mention below will be left to the developer using the actor platform
since not all problems can be solved in a generic way by the
platform.

What I want is to leave that kind of decisions to the user. For this
to work the user needs to be able to make changes to the way the
queues are set up, which means that the actor platform needs to be
sufficiently configurable for this. What I'm thinking of is to allow
the user to specify the queue to be used for every actor. Maybe, by
default, the single default queue is always used if not specified
differently. So the user can add queues in case it seems appropriate.
Being able to add queues on demand means that message order must not
be assumed to be retained by the recipient, though.

> So the big question is what those consumers do with the payload? Are those
> thousands of consumers running on the same system, or jvm even?  Then, do
> the business logic in the consumers hog network, memory, cpu time, database
> resources? Is the system able to cope with that kind of concurrent fight
> over system resourses? It might be a problem, it might not, but every system
> has its bottlenecks.
>
> What I'm trying to say is that just consuming lots of messages concurrently
> by itself isn't hard. It is when you try to do something interesting with
> those messages that you're likely to run into performance issuses.

Yes, that's right. I see. I see no way round this than recommending
the user to avoid long-runners. Another approach is to make actors
interruptible. Since a thread cannot be suspended and resumed in Java,
some synchronized interruptCount flag would be increment in case an
asynchronous event arrives and the actor during its execution would
have to poll this flag from time to time.

> >  1) 1 JMS publish-subscriber channel with one dispatcher at the end
> >  that takes every msg from the channel and propagates it to the
> >  respective consumer out of those thousand consumers.
>
> Each endpoint consumer can consume directly from the jms. What purpose would
> in-between home grown dispatcher have? Consuming messages isn't expensive,
> and a dispatcher might turn out to be a bottleneck in the system. I'd call 1
> a antipattern.

When an actor is busy consuming a message it cannot respond to
external asynchronous events. In case one arrives the thread the
dispatcher runs in would pass the event on to the actor and increment
the actor's interruptCount flag. Once it is done with that, it
continues to dispatch messages.

> >  2) 1 JMS publish-subscriber channel where every consumer peeks a newly
> >  arrived msg and checks whether it is for him and takes it from the
> >  queue if so.
>
> If you choose path 2, you'll want to look into message selectors and
> flagging each message with heades so consumers that should not grab the
> message doesn't even see the message. You do not want a jms consumer to
> reject a dispatched message when the message cannot be redelivered to the
> same consumer at a later time. Again a jms antipattern.
>
> I'd caution against message selectors when your consumers can predict
> which destination to poll. Using a dedicated destination - queue or topic
> name - for each message type is more efficient when the consumers doesn't
> have to consume several different message types.
>
> >  3) Many JMS publish-subscriber channels with one dispatcher for
> >  everyone. I favour this one.
>
> >  4) Thousand point-to-point connections. Sure not, I guess.
>
> I get the impression that your messages are unique and should only be
> consumed once by a single consumer? If that's correct you should use the
> point-to-point queue based jms-pattern in 4, rather than using pub-sub with
> topics in 3. You'll only want to use topics when the same message should be
> dispatched to multiple consumers.

Thanks Fredrik. What you explain above is very useful.

I will start a new job in some few months where I will also work with
ActiveMQ. Maybe, I keep things in vm-local mode with my little actor
framework using JDK1.5 BlockingQueues instead of JMS queues and will
lateron add an extension to it using JMS queues for distributed actors
once I have learned new things about JMS in my new job. Problem is
some fear that just adding JMS queues to it later will turn out as not
feasible since basic assumptions made in the vm-local first approach
turn out not to be valid any more with when the whole thing get
distributed... That's why I started this thread.

Regards, Oliver

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

JMS scalability question saxo123@gmx.de - 2011-11-15 02:07 -0800
  Re: JMS scalability question Roedy Green <see_website@mindprod.com.invalid> - 2011-11-16 06:21 -0800
    Re: JMS scalability question saxo123@gmx.de - 2011-11-16 09:35 -0800
      Re: JMS scalability question Lew <lewbloch@gmail.com> - 2011-11-16 10:30 -0800
        Re: JMS scalability question saxo123@gmx.de - 2011-11-16 14:25 -0800
          Re: JMS scalability question Lew <lewbloch@gmail.com> - 2011-11-16 17:18 -0800
            Re: JMS scalability question saxo123@gmx.de - 2011-11-16 22:59 -0800
              Re: JMS scalability question jlp <jlp@jlp.com> - 2011-11-17 08:41 +0100
                Re: JMS scalability question saxo123@gmx.de - 2011-11-17 01:00 -0800
      Re: JMS scalability question Roedy Green <see_website@mindprod.com.invalid> - 2011-11-16 16:16 -0800
  Re: JMS scalability question Fredrik Jonson <fredrik@jonson.org> - 2011-11-17 08:11 +0000
    Re: JMS scalability question saxo123@gmx.de - 2011-11-17 01:26 -0800
      Re: JMS scalability question Fredrik Jonson <fredrik@jonson.org> - 2011-11-17 21:01 +0000
        Re: JMS scalability question Saxo <saxo123@gmx.de> - 2011-11-17 23:27 -0800

csiph-web