Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming.threads > #449 > unrolled thread

connection thread bottleneck due to contention on global data

Back to article view | Back to comp.programming.threads

Contents

  connection thread bottleneck due to contention on global data itt ium <ittium@gmail.com> - 2011-07-16 05:31 -0700
    Re: connection thread bottleneck due to contention on global data David Schwartz <davids@webmaster.com> - 2011-07-16 11:49 -0700
      Re: connection thread bottleneck due to contention on global data itt ium <ittium@gmail.com> - 2011-07-16 23:08 -0700
        Re: connection thread bottleneck due to contention on global data David Schwartz <davids@webmaster.com> - 2011-07-17 00:00 -0700
          Re: connection thread bottleneck due to contention on global data ittium <ittium@gmail.com> - 2011-07-17 13:29 +0530
            Re: connection thread bottleneck due to contention on global data David Schwartz <davids@webmaster.com> - 2011-07-17 08:30 -0700
              Re: connection thread bottleneck due to contention on global data itt ium <ittium@gmail.com> - 2011-07-18 00:43 -0700
                Re: connection thread bottleneck due to contention on global data David Schwartz <davids@webmaster.com> - 2011-07-18 10:41 -0700
                  Re: connection thread bottleneck due to contention on global data frege <gottlobfrege@gmail.com> - 2011-07-20 22:18 -0700
    Re: connection thread bottleneck due to contention on global data "Chris M. Thomasson" <cristom@charter.net> - 2011-07-17 16:43 -0700
      Re: connection thread bottleneck due to contention on global data itt ium <ittium@gmail.com> - 2011-07-18 00:44 -0700
        Re: connection thread bottleneck due to contention on global data "Chris M. Thomasson" <cristom@charter.net> - 2011-07-23 23:18 -0700

#449 — connection thread bottleneck due to contention on global data

Group,
I discussed the similar issue couple of month back in this group. Your
inputs helped me a lot to fix parts of my design. I would like to hear
from you now on following issue.

I have a situation where I have large number of worker thread (~10)
acting as consumer and good number of connection thread (acting as
producer). The system is processing (LTE) network packet. When first
message for a call is received, it is assigned to one of the worker
thread in round robin  fashion and an entry is made in global t
"callId vs worker thread id" map/hash map. Subsequent message of the
call(that may come on any connection) are also assigned to same worker
thread by consulting the
"callId vs worker thread id" map/hash map.

When call is terminated entry from the map is removed.

Each worker thread has its own dedicated database using which it can
process the call message and no locks are needed.

My problem is regarding this global "callId vs worker thread id" map/
hash map that is protected by mutex. Assuming futex based mutex
implementation, at high load, it is likely there will be large number
of contention , making the connection thread sleep and wake up large
number of times. A spinlock based implementation is unlikely to solve
this situation. This situation is made more complex by the fact that
to find out the call Id, connection thread will have to decode the
message.

Lot of processing on connection thread and sleep/wake up on mutex is
likely to affect connection protocol e.g in TCP recv buffer may get
full since application is not able to remove data from the socket recv
buffer.

I am assuming many guys on this groups might have designed similar
system. Please comment on above design.

thanks
Ittium

[toc] | [next] | [standalone]

#450

On Jul 16, 5:31 am, itt ium <itt...@gmail.com> wrote:

> My problem is regarding this global "callId vs worker thread id" map/
> hash map that is protected by mutex. Assuming futex based mutex
> implementation, at high load, it is likely there will be large number
> of contention , making the connection thread sleep and wake up large
> number of times. A spinlock based implementation is unlikely to solve
> this situation. This situation is made more complex by the fact that
> to find out the call Id, connection thread will have to decode the
> message.
>
> Lot of processing on connection thread and sleep/wake up on mutex is
> likely to affect connection protocol e.g in TCP recv buffer may get
> full since application is not able to remove data from the socket recv
> buffer.

I think the problem you are worrying about can't possibly happen. Each
thread needs this mutex for only the most miniscule fraction of their
running time. Why would you expect enough contention to be a problem?

DS

[toc] | [prev] | [next] | [standalone]

#451

On Jul 16, 11:49 pm, David Schwartz <dav...@webmaster.com> wrote:
> On Jul 16, 5:31 am, itt ium <itt...@gmail.com> wrote:
>
> > My problem is regarding this global "callId vs worker thread id" map/
> > hash map that is protected by mutex. Assuming futex based mutex
> > implementation, at high load, it is likely there will be large number
> > of contention , making the connection thread sleep and wake up large
> > number of times. A spinlock based implementation is unlikely to solve
> > this situation. This situation is made more complex by the fact that
> > to find out the call Id, connection thread will have to decode the
> > message.
>
> > Lot of processing on connection thread and sleep/wake up on mutex is
> > likely to affect connection protocol e.g in TCP recv buffer may get
> > full since application is not able to remove data from the socket recv
> > buffer.
>
> I think the problem you are worrying about can't possibly happen. Each
> thread needs this mutex for only the most miniscule fraction of their
> running time. Why would you expect enough contention to be a problem?
>
> DS

thanks DS for the response
I was going through the futex based mutex implementation on linux. All
non contention cases will be resolved at user level there is no user-
kernel mode switching. In my case, at a high load, at any point of
time, contention is likely to be always resolved by kernel, since
kernel can give the mutex to only one thread other thread will sleep
possibly for a very small amount of time. I am not sure about cost of
this usermode<->kernel mode switching and large number of sleep and
wakeup.

[toc] | [prev] | [next] | [standalone]

#452

On Jul 16, 11:08 pm, itt ium <itt...@gmail.com> wrote:

> I was going through the futex based mutex implementation on linux. All
> non contention cases will be resolved at user level there is no user-
> kernel mode switching. In my case, at a high load, at any point of
> time, contention is likely to be always resolved by kernel, since
> kernel can give the mutex to only one thread other thread will sleep
> possibly for a very small amount of time. I am not sure about cost of
> this usermode<->kernel mode switching and large number of sleep and
> wakeup.

Why do you think that will happen? All of your threads hold the mutex
for an insignificant fraction of their execution time, right? If
that's not so, then *that's* your issue.

DS

[toc] | [prev] | [next] | [standalone]

#453

>
> Why do you think that will happen? All of your threads hold the mutex
> for an insignificant fraction of their execution time, right? If
> that's not so, then *that's* your issue.
>
Please read my earlier comment again, if more than one thread is waiting 
for a mutex then kernel
can give the mutex to only one thread other threads will be asked to 
sleep (although for a very small time). It give me the impression that 
mutex should be avoided for data with moderate contention and should be 
used only when contention is really low.

[toc] | [prev] | [next] | [standalone]

#454

On Jul 17, 12:59 am, ittium <itt...@gmail.com> wrote:

> Please read my earlier comment again, if more than one thread is waiting
> for a mutex then kernel
> can give the mutex to only one thread other threads will be asked to
> sleep (although for a very small time). It give me the impression that
> mutex should be avoided for data with moderate contention and should be
> used only when contention is really low.

In this case contention is nearly zero. Each thread only holds the
mutex for the most insignificant fraction of its run time. Is that
correct?

DS

[toc] | [prev] | [next] | [standalone]

#458

>
> In this case contention is nearly zero. Each thread only holds the
> mutex for the most insignificant fraction of its run time. Is that
> correct?
>

DS, Please correct me If I am wrong. when you say contention is high,
you mean that a shared data is accessed for a long period of time and
good number of thread are likely to be waiting for it at any point of
time but if mutex hold time is small, contention is assumed to be
small even if many threads are likely to be waiting for the mutex at
any point of time. I am really concerned if there is a single mutex
and more than one threads are waiting for it to become free, only one
will get it and other will probably be asked by kernel to sleep. I
fear if this situation is common in a design, it will affect the
performance.

[toc] | [prev] | [next] | [standalone]

#463

On Jul 18, 12:43 am, itt ium <itt...@gmail.com> wrote:

> DS, Please correct me If I am wrong. when you say contention is high,
> you mean that a shared data is accessed for a long period of time and
> good number of thread are likely to be waiting for it at any point of
> time but if mutex hold time is small, contention is assumed to be
> small even if many threads are likely to be waiting for the mutex at
> any point of time. I am really concerned if there is a single mutex
> and more than one threads are waiting for it to become free, only one
> will get it and other will probably be asked by kernel to sleep. I
> fear if this situation is common in a design, it will affect the
> performance.

Why are you worried about this issue with this mutex though? It seems
baffling. If you have four threads each of which need a mutex for 1%
of their operating time, there will be effectively on contention at
all. It won't matter.

DS

[toc] | [prev] | [next] | [standalone]

#469

On Jul 18, 1:41 pm, David Schwartz <dav...@webmaster.com> wrote:
> On Jul 18, 12:43 am, itt ium <itt...@gmail.com> wrote:
>
>
> Why are you worried about this issue with this mutex though? It seems
> baffling. If you have four threads each of which need a mutex for 1%
> of their operating time, there will be effectively on contention at
> all. It won't matter.
>
> DS

In other words:

Have you *measured* that your mutex is a problem?

Knuth premature something something.

Tony

[toc] | [prev] | [next] | [standalone]

#456

"itt ium" <ittium@gmail.com> wrote in message 
news:85f02a56-5aaa-4d6d-a4ef-5f880f3f67cc@j9g2000prj.googlegroups.com...
> Group,
> I discussed the similar issue couple of month back in this group. Your
> inputs helped me a lot to fix parts of my design. I would like to hear
> from you now on following issue.
>
> I have a situation where I have large number of worker thread (~10)
> acting as consumer and good number of connection thread (acting as
> producer). The system is processing (LTE) network packet. When first
> message for a call is received, it is assigned to one of the worker
> thread in round robin  fashion and an entry is made in global t
> "callId vs worker thread id" map/hash map. Subsequent message of the
> call(that may come on any connection) are also assigned to same worker
> thread by consulting the
> "callId vs worker thread id" map/hash map.

[...]

Are you using a single mutex to protect this global hash map? Can you get 
away with a mutex per-bucket? 

[toc] | [prev] | [next] | [standalone]

#461

>
> [...]
>
> Are you using a single mutex to protect this global hash map? Can you get
> away with a mutex per-bucket?

Chris, I am using C++ standard hash map, I am not sure if this is
possible, Can you give some example or link.
thanks
Ittium

[toc] | [prev] | [next] | [standalone]

#472

"itt ium" <ittium@gmail.com> wrote in message 
news:6088bc05-4bf5-489e-ba1b-7e949dc7fd46@r11g2000prd.googlegroups.com...
>
>>
>> [...]
>>
>> Are you using a single mutex to protect this global hash map? Can you get
>> away with a mutex per-bucket?
>
> Chris, I am using C++ standard hash map, I am not sure if this is
> possible, Can you give some example or link.

I don't think it's possible. You would have to create you're own hash map 
that has finer grain synchronization. Or, perhaps use some "off the shelf" 
packages... 

[toc] | [prev] | [standalone]

Back to top | Article view | comp.programming.threads

csiph-web