Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #146382 > unrolled thread

realloc() and POSIX threads

Back to article view | Back to comp.lang.c

Contents

  realloc() and POSIX threads Doug Weiman <no@nyet.net> - 2019-09-27 21:29 +0000
    Re: realloc() and POSIX threads Robert Wessel <robertwessel2@yahoo.com> - 2019-09-27 17:11 -0500
    Re: realloc() and POSIX threads Florian Weimer <fw@deneb.enyo.de> - 2019-09-28 09:27 +0200
    Re: realloc() and POSIX threads Melzzzzz <Melzzzzz@zzzzz.com> - 2019-09-28 08:42 +0000
    Re: realloc() and POSIX threads "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2019-09-28 03:53 -0700
      Re: realloc() and POSIX threads "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2019-09-28 03:55 -0700
    Re: realloc() and POSIX threads Richard Damon <Richard@Damon-Family.org> - 2019-09-28 08:29 -0400
      Re: realloc() and POSIX threads Doug Weiman <no@nyet.net> - 2019-09-28 15:06 +0000
      Re: realloc() and POSIX threads "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2019-09-28 15:19 -0700
        Re: realloc() and POSIX threads "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2019-09-28 15:23 -0700

#146382 — realloc() and POSIX threads

	My question may not belong in this group - my apologies if that 
is the case.

	I have some application that allocates memory with malloc(), and 
then uses the resulting pointer as an argument to a POSIX thread that is 
launched immediately after. While the thread is doing its thing with the 
memory pointed to by this pointer, the application augments that pointer 
with realloc().

	My understanding is that realloc() is no guaranteed to keep the 
original pointer. That is, if necessary, realloc() will obtain a 
different pointer to a memory region with enough space, and copy here the 
contents of the memory region pointed to by the original pointer, finally 
returning thee new pointer to the caller.

	This is, of course, potentially disastrous to the thread above, 
which is still operating on a memory area now deallocated. I mean, you 
may be lucky and the area will still be usable, but sooner or later the 
whole thing will blow up in yor face.

	Is there a way to implement the scenario above (memory 
allocation, thread creation, memory reallocation) safely, so that the 
thread can be guaranteed to still have access to valid memory? One could 
use thread mutexes, but that would effectively serialize things.

	Any other suggestions? Or is this the kind of thing that one just 
ought not to do, period? For a number of reasons, getting the thread 
itself to allocate the memory is not an option.

[toc] | [next] | [standalone]

#146383

On Fri, 27 Sep 2019 21:29:47 +0000 (UTC), Doug Weiman <no@nyet.net>
wrote:

>	My question may not belong in this group - my apologies if that 
>is the case.
>
>	I have some application that allocates memory with malloc(), and 
>then uses the resulting pointer as an argument to a POSIX thread that is 
>launched immediately after. While the thread is doing its thing with the 
>memory pointed to by this pointer, the application augments that pointer 
>with realloc().
>
>	My understanding is that realloc() is no guaranteed to keep the 
>original pointer. That is, if necessary, realloc() will obtain a 
>different pointer to a memory region with enough space, and copy here the 
>contents of the memory region pointed to by the original pointer, finally 
>returning thee new pointer to the caller.
>
>	This is, of course, potentially disastrous to the thread above, 
>which is still operating on a memory area now deallocated. I mean, you 
>may be lucky and the area will still be usable, but sooner or later the 
>whole thing will blow up in yor face.
>
>	Is there a way to implement the scenario above (memory 
>allocation, thread creation, memory reallocation) safely, so that the 
>thread can be guaranteed to still have access to valid memory? One could 
>use thread mutexes, but that would effectively serialize things.
>
>	Any other suggestions? Or is this the kind of thing that one just 
>ought not to do, period? For a number of reasons, getting the thread 
>itself to allocate the memory is not an option.



Not really, things that can be changed by multiple threads, need to be
appropriately protected.  In this case you have a second thread with a
copy of a pointer to an area, and the actual pointer get changed, and
there's nothing to get the copy updated (which might just be in a
loaded into register in the other thread).  Even worse, even if there
was a way to do that, the copy operation that realloc is doing might
be copying the pointed-to structure as that structure is being updated
by the other thread, thus resulting in the new area having only part
of the update.

The obvious solution is to protect all uses of this pointer with a
mutex.  A read/write (or shared/exclusive) mutex can be useful if
actual updates of the pointer are rare, and multiple accesses to the
object being pointed to are.

Another possibility is versioning of some sort.  Don't immediately
replace the object, rather create a new one, and start using that,
while existing work continues using the old object.  That may, or may
not, be appropriate for a particular application, and does add some
complications about just when the object gets freed.

[toc] | [prev] | [next] | [standalone]

#146386

* Doug Weiman:

> 	Is there a way to implement the scenario above (memory 
> allocation, thread creation, memory reallocation) safely, so that the 
> thread can be guaranteed to still have access to valid memory? One could 
> use thread mutexes, but that would effectively serialize things.

There are transactional memory techniques that can avoid atomic writes
on the reader side, but they involve a custom memory allocator that
ensures that access to the memory region never has side effects (even
after deallocation), and they also need widely implemented compiler
extensions to avoid undefined behavior due to the data races and
aliasing violations.

Read-copy-update is another technique that can avoid synchronization
costs.

[toc] | [prev] | [next] | [standalone]

#146388

On 2019-09-27, Doug Weiman <no@nyet.net> wrote:
> 	My question may not belong in this group - my apologies if that 
> is the case.
>
> 	I have some application that allocates memory with malloc(), and 
> then uses the resulting pointer as an argument to a POSIX thread that is 
> launched immediately after. While the thread is doing its thing with the 
> memory pointed to by this pointer, the application augments that pointer 
> with realloc().
>
> 	My understanding is that realloc() is no guaranteed to keep the 
> original pointer. That is, if necessary, realloc() will obtain a 
> different pointer to a memory region with enough space, and copy here the 
> contents of the memory region pointed to by the original pointer, finally 
> returning thee new pointer to the caller.
>
> 	This is, of course, potentially disastrous to the thread above, 
> which is still operating on a memory area now deallocated. I mean, you 
> may be lucky and the area will still be usable, but sooner or later the 
> whole thing will blow up in yor face.
>
> 	Is there a way to implement the scenario above (memory 
> allocation, thread creation, memory reallocation) safely, so that the 
> thread can be guaranteed to still have access to valid memory? One could 
> use thread mutexes, but that would effectively serialize things.
>
> 	Any other suggestions? Or is this the kind of thing that one just 
> ought not to do, period? For a number of reasons, getting the thread 
> itself to allocate the memory is not an option.
I don't know what you ask? Calling realloc reuires all pointers to be
updated anyway...

-- 
press any key to continue or any other to quit...
U ničemu ja ne uživam kao u svom statusu INVALIDA -- Zli Zec
Na divljem zapadu i nije bilo tako puno nasilja, upravo zato jer su svi 
bili naoruzani. -- Mladen Gogala

[toc] | [prev] | [next] | [standalone]

#146389

On 9/27/2019 2:29 PM, Doug Weiman wrote:
> 	My question may not belong in this group - my apologies if that
> is the case.
> 
> 	I have some application that allocates memory with malloc(), and
> then uses the resulting pointer as an argument to a POSIX thread that is
> launched immediately after. While the thread is doing its thing with the
> memory pointed to by this pointer, the application augments that pointer
> with realloc().
> 
> 	My understanding is that realloc() is no guaranteed to keep the
> original pointer. That is, if necessary, realloc() will obtain a
> different pointer to a memory region with enough space, and copy here the
> contents of the memory region pointed to by the original pointer, finally
> returning thee new pointer to the caller.
> 
> 	This is, of course, potentially disastrous to the thread above,
> which is still operating on a memory area now deallocated. I mean, you
> may be lucky and the area will still be usable, but sooner or later the
> whole thing will blow up in yor face.
> 
> 	Is there a way to implement the scenario above (memory
> allocation, thread creation, memory reallocation) safely, so that the
> thread can be guaranteed to still have access to valid memory? One could
> use thread mutexes, but that would effectively serialize things.
> 
> 	Any other suggestions? Or is this the kind of thing that one just
> ought not to do, period? For a number of reasons, getting the thread
> itself to allocate the memory is not an option.
> 

Never call realloc on live memory, where other threads can be using it. 
Instead, wait for the memory to go into a quiescent state, then you can 
free it, and call realloc.

[toc] | [prev] | [next] | [standalone]

#146390

On 9/28/2019 3:53 AM, Chris M. Thomasson wrote:
> On 9/27/2019 2:29 PM, Doug Weiman wrote:
>>     My question may not belong in this group - my apologies if that
>> is the case.
>>
>>     I have some application that allocates memory with malloc(), and
>> then uses the resulting pointer as an argument to a POSIX thread that is
>> launched immediately after. While the thread is doing its thing with the
>> memory pointed to by this pointer, the application augments that pointer
>> with realloc().
>>
>>     My understanding is that realloc() is no guaranteed to keep the
>> original pointer. That is, if necessary, realloc() will obtain a
>> different pointer to a memory region with enough space, and copy here the
>> contents of the memory region pointed to by the original pointer, finally
>> returning thee new pointer to the caller.
>>
>>     This is, of course, potentially disastrous to the thread above,
>> which is still operating on a memory area now deallocated. I mean, you
>> may be lucky and the area will still be usable, but sooner or later the
>> whole thing will blow up in yor face.
>>
>>     Is there a way to implement the scenario above (memory
>> allocation, thread creation, memory reallocation) safely, so that the
>> thread can be guaranteed to still have access to valid memory? One could
>> use thread mutexes, but that would effectively serialize things.
>>
>>     Any other suggestions? Or is this the kind of thing that one just
>> ought not to do, period? For a number of reasons, getting the thread
>> itself to allocate the memory is not an option.
>>
> 
> Never call realloc on live memory, where other threads can be using it. 
> Instead, wait for the memory to go into a quiescent state, then you can 
> free it, and call realloc.
         ^^^^^^^^^^

typo.

free it, reuse it, call realloc, whatever. Since realloc can act like a 
free, well, you must ensure that no other thread is using the memory. 
There are many techniques to accomplish this.

[toc] | [prev] | [next] | [standalone]

#146392

On 9/27/19 5:29 PM, Doug Weiman wrote:
> 	My question may not belong in this group - my apologies if that 
> is the case.
> 
> 	I have some application that allocates memory with malloc(), and 
> then uses the resulting pointer as an argument to a POSIX thread that is 
> launched immediately after. While the thread is doing its thing with the 
> memory pointed to by this pointer, the application augments that pointer 
> with realloc().
> 
> 	My understanding is that realloc() is no guaranteed to keep the 
> original pointer. That is, if necessary, realloc() will obtain a 
> different pointer to a memory region with enough space, and copy here the 
> contents of the memory region pointed to by the original pointer, finally 
> returning thee new pointer to the caller.
> 
> 	This is, of course, potentially disastrous to the thread above, 
> which is still operating on a memory area now deallocated. I mean, you 
> may be lucky and the area will still be usable, but sooner or later the 
> whole thing will blow up in yor face.
> 
> 	Is there a way to implement the scenario above (memory 
> allocation, thread creation, memory reallocation) safely, so that the 
> thread can be guaranteed to still have access to valid memory? One could 
> use thread mutexes, but that would effectively serialize things.
> 
> 	Any other suggestions? Or is this the kind of thing that one just 
> ought not to do, period? For a number of reasons, getting the thread 
> itself to allocate the memory is not an option.
> 

Pass the thread a POINTER to the pointer to the memory, that way when
the pointer to memory is update, the thread can see the change.

You will still need some protection (like with a mutex) that you don't
update the pointer at a period when the block is actively being used,
and use the appropriate memory barriers so that the change is properly seen.

[toc] | [prev] | [next] | [standalone]

#146395

On Sat, 28 Sep 2019 08:29:04 -0400, Richard Damon wrote:

> On 9/27/19 5:29 PM, Doug Weiman wrote:
>> 	My question may not belong in this group - my apologies if that
>> is the case.
>> 
>> 	I have some application that allocates memory with malloc(), and
>> then uses the resulting pointer as an argument to a POSIX thread that
>> is launched immediately after. While the thread is doing its thing with
>> the memory pointed to by this pointer, the application augments that
>> pointer with realloc().
>> 
>> 	My understanding is that realloc() is no guaranteed to keep the
>> original pointer. That is, if necessary, realloc() will obtain a
>> different pointer to a memory region with enough space, and copy here
>> the contents of the memory region pointed to by the original pointer,
>> finally returning thee new pointer to the caller.
>> 
>> 	This is, of course, potentially disastrous to the thread above,
>> which is still operating on a memory area now deallocated. I mean, you
>> may be lucky and the area will still be usable, but sooner or later the
>> whole thing will blow up in yor face.
>> 
>> 	Is there a way to implement the scenario above (memory
>> allocation, thread creation, memory reallocation) safely, so that the
>> thread can be guaranteed to still have access to valid memory? One
>> could use thread mutexes, but that would effectively serialize things.
>> 
>> 	Any other suggestions? Or is this the kind of thing that one just
>> ought not to do, period? For a number of reasons, getting the thread
>> itself to allocate the memory is not an option.
>> 
>> 
> Pass the thread a POINTER to the pointer to the memory, that way when
> the pointer to memory is update, the thread can see the change.

	Good thinking - I'll definitely try that.

> You will still need some protection (like with a mutex) that you don't
> update the pointer at a period when the block is actively being used,
> and use the appropriate memory barriers so that the change is properly
> seen.

[toc] | [prev] | [next] | [standalone]

#146403

On 9/28/2019 5:29 AM, Richard Damon wrote:
> On 9/27/19 5:29 PM, Doug Weiman wrote:
>> 	My question may not belong in this group - my apologies if that
>> is the case.
[...]
> Pass the thread a POINTER to the pointer to the memory, that way when
> the pointer to memory is update, the thread can see the change.
> You will still need some protection (like with a mutex) that you don't
> update the pointer at a period when the block is actively being used,

Its better to not have the updates depend on user activity. The reads 
and writes are separated, and do not depend on each others activity. 
There can be updates while readers are reading.

> and use the appropriate memory barriers so that the change is properly seen.
> 

RCU is a method for allowing writers and readers to run in parallel. 
Writers do not need to wait for any reader activity, and vise versa. 
There is no sync between reads and writes. Reads run full steam ahead. 
Great for read mostly data-structures.

[toc] | [prev] | [next] | [standalone]

#146404

On 9/28/2019 3:19 PM, Chris M. Thomasson wrote:
> On 9/28/2019 5:29 AM, Richard Damon wrote:
>> On 9/27/19 5:29 PM, Doug Weiman wrote:
>>>     My question may not belong in this group - my apologies if that
>>> is the case.
> [...]
>> Pass the thread a POINTER to the pointer to the memory, that way when
>> the pointer to memory is update, the thread can see the change.
>> You will still need some protection (like with a mutex) that you don't
>> update the pointer at a period when the block is actively being used,
> 
> Its better to not have the updates depend on user activity. The reads 
> and writes are separated, and do not depend on each others activity. 
> There can be updates while readers are reading.
> 
>> and use the appropriate memory barriers so that the change is properly 
>> seen.
>>
> 
> RCU is a method for allowing writers and readers to run in parallel. 
> Writers do not need to wait for any reader activity, and vise versa. 
> There is no sync between reads and writes. Reads run full steam ahead. 
> Great for read mostly data-structures.

There is a way to code a user-space version of this up using pure C11. 
Its not quite as fast as native RCU, but beats the hell out of a 
read-write mutex. The usage pattern is geared towards read-mostly data 
structures.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web