Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #12911 > unrolled thread

C++ Memory Management Question

Back to article view | Back to comp.lang.c++

Contents

  C++ Memory Management Question Datesfat Chicks <datesfat.chicks@gmail.com> - 2011-12-31 08:56 -0500
    Re: C++ Memory Management Question Victor Bazarov <v.bazarov@comcast.invalid> - 2011-12-31 09:44 -0500
      Re: C++ Memory Management Question Datesfat Chicks <datesfat.chicks@gmail.com> - 2011-12-31 10:17 -0500
        Re: C++ Memory Management Question "MikeWhy" <boat042-nospam@yahoo.com> - 2011-12-31 12:56 -0600
        Re: C++ Memory Management Question jt@toerring.de (Jens Thoms Toerring) - 2011-12-31 19:05 +0000
      Re: C++ Memory Management Question Carlo Milanesi <carlo.nospam.milanesi@libero.it> - 2011-12-31 18:18 +0100
        Re: C++ Memory Management Question Victor Bazarov <v.bazarov@comcast.invalid> - 2011-12-31 13:04 -0500
          Re: C++ Memory Management Question Carlo Milanesi <carlo.nospam.milanesi@libero.it> - 2011-12-31 19:42 +0100
          Re: C++ Memory Management Question Goran <goran.pusic@gmail.com> - 2011-12-31 23:25 -0800
            Re: C++ Memory Management Question Victor Bazarov <v.bazarov@comcast.invalid> - 2012-01-01 09:39 -0500
              Re: C++ Memory Management Question Goran <goran.pusic@gmail.com> - 2012-01-01 08:20 -0800
              Re: C++ Memory Management Question Leigh Johnston <leigh@i42.co.uk> - 2012-01-01 19:05 +0000
                Re: C++ Memory Management Question "Paul" <pchrist<nospam>or@yahoo.co.uk> - 2012-01-01 19:36 +0000
                  Re: C++ Memory Management Question Leigh Johnston <leigh@i42.co.uk> - 2012-01-01 19:49 +0000
                    Re: C++ Memory Management Question "Paul" <pchrist<nospam>or@yahoo.co.uk> - 2012-01-02 11:26 +0000
                      Re: C++ Memory Management Question Leigh Johnston <leigh@i42.co.uk> - 2012-01-02 14:32 +0000
                        Re: C++ Memory Management Question "Paul" <pchrist<nospam>or@yahoo.co.uk> - 2012-01-02 15:54 +0000
                          Re: C++ Memory Management Question Leigh Johnston <leigh@i42.co.uk> - 2012-01-02 16:26 +0000
                            Re: C++ Memory Management Question "Paul" <pchrist<nospam>or@yahoo.co.uk> - 2012-01-02 16:58 +0000
                              Re: C++ Memory Management Question Leigh Johnston <leigh@i42.co.uk> - 2012-01-02 21:14 +0000
                                Re: C++ Memory Management Question Ian Collins <ian-news@hotmail.com> - 2012-01-03 11:11 +1300
                                Re: C++ Memory Management Question scott@slp53.sl.home (Scott Lurndal) - 2012-01-03 20:53 +0000
                                  Re: C++ Memory Management Question Leigh Johnston <leigh@i42.co.uk> - 2012-01-03 21:21 +0000
                                  Re: C++ Memory Management Question Ian Collins <ian-news@hotmail.com> - 2012-01-04 12:15 +1300
                                Re: C++ Memory Management Question gwowen <gwowen@gmail.com> - 2012-01-04 08:20 -0800
                            Re: C++ Memory Management Question "Paul" <pchrist<nospam>or@yahoo.co.uk> - 2012-01-02 22:24 +0000
                              Re: C++ Memory Management Question Leigh Johnston <leigh@i42.co.uk> - 2012-01-02 22:55 +0000
                                Re: C++ Memory Management Question "Paul" <pchrist<nospam>or@yahoo.co.uk> - 2012-01-02 23:27 +0000
                                Re: C++ Memory Management Question Nick Keighley <nick_keighley_nospam@hotmail.com> - 2012-01-10 01:01 -0800
                        Re: C++ Memory Management Question 88888 Dihedral <dihedral88888@googlemail.com> - 2012-01-02 21:45 -0800
                          Re: C++ Memory Management Question Goran <goran.pusic@gmail.com> - 2012-01-03 00:32 -0800
                            Re: C++ Memory Management Question "Paul" <pchrist<nospam>or@yahoo.co.uk> - 2012-01-03 17:34 +0000
                        Re: C++ Memory Management Question Nick Keighley <nick_keighley_nospam@hotmail.com> - 2012-01-10 00:47 -0800
                          Re: C++ Memory Management Question BCFD36 <bcfd36@cruzio.com> - 2012-01-10 22:57 -0800
                Re: C++ Memory Management Question Ian Collins <ian-news@hotmail.com> - 2012-01-03 09:31 +1300
                  Re: C++ Memory Management Question Leigh Johnston <leigh@i42.co.uk> - 2012-01-02 21:02 +0000
                    Re: C++ Memory Management Question Ian Collins <ian-news@hotmail.com> - 2012-01-03 11:09 +1300
            Re: C++ Memory Management Question Leigh Johnston <leigh@i42.co.uk> - 2012-01-01 18:57 +0000
        Re: C++ Memory Management Question jgk@panix.com (Joe keane) - 2012-01-01 22:04 +0000
          Re: C++ Memory Management Question Carlo Milanesi <carlo.nospam.milanesi@libero.it> - 2012-01-02 00:37 +0100
            Re: C++ Memory Management Question yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2012-01-09 17:41 +0000
              Re: C++ Memory Management Question Carlo Milanesi <carlo.milanesi@libero.it> - 2012-01-13 08:14 -0800
      Re: C++ Memory Management Question Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-01-01 18:05 +0000
    Re: C++ Memory Management Question Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-12-31 07:25 -0800
    Re: C++ Memory Management Question Torsten Mueller <dev-null@shared-files.de> - 2011-12-31 16:41 +0100
    Re: C++ Memory Management Question Nobody <nobody@nowhere.com> - 2012-01-01 09:16 +0000
      Re: C++ Memory Management Question Datesfat Chicks <datesfat.chicks@gmail.com> - 2012-01-02 16:48 -0500
        Re: C++ Memory Management Question Goran <goran.pusic@gmail.com> - 2012-01-03 00:02 -0800
          Re: C++ Memory Management Question Datesfat Chicks <datesfat.chicks@gmail.com> - 2012-01-03 12:53 -0500
            Re: C++ Memory Management Question Nobody <nobody@nowhere.com> - 2012-01-03 19:46 +0000
            Re: C++ Memory Management Question Goran <goran.pusic@gmail.com> - 2012-01-04 00:11 -0800
              Re: C++ Memory Management Question Nobody <nobody@nowhere.com> - 2012-01-04 09:30 +0000
      Re: C++ Memory Management Question Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-01-04 19:28 +0000
        Re: C++ Memory Management Question Nobody <nobody@nowhere.com> - 2012-01-04 22:09 +0000
          Re: C++ Memory Management Question Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-01-05 11:43 +0000
    Re: C++ Memory Management Question Christian Gollwitzer <auriocus@gmx.de> - 2012-01-01 18:13 +0100
      Re: C++ Memory Management Question "io_x" <a@b.c.invalid> - 2012-01-01 20:11 +0100
      Re: C++ Memory Management Question Datesfat Chicks <datesfat.chicks@gmail.com> - 2012-01-02 16:51 -0500
    Re: C++ Memory Management Question Simon Quinn <sq@nospam.com> - 2012-01-05 07:54 +0000
    Re: C++ Memory Management Question darylew@gmail.com - 2012-01-10 10:19 -0800

Page 3 of 3 — ← Prev page 1 2 [3]

#13188

In article <4f00ee2d$0$6836$5fc30a8@news.tiscali.it>,
Carlo Milanesi  <carlo.nospam.milanesi@libero.it> wrote:
>On 01/01/2012 23:04, Joe keane wrote:
>> In article<4eff43fa$0$16659$5fc30a8@news.tiscali.it>,
>> Carlo Milanesi<carlo.nospam.milanesi@libero.it>  wrote:
>>> So, if your word processor cannot load a huge document, or your video
>>> editor cannot load a huge piece of footage, you'd like the application
>>> terminate at once without saving anything :(
>>
>> no it could produce a useful measage
>>
>> a) your hard drive is full
>> b) this can't possibly work on your machine
>> c) we ran into VM problems, if you close something else, maybe it will work
>
>I was sarcastic, of course.
>However those messages are not really informative.
>Microsoft Paint ("prush.exe", the bitmap editing utility included in 
>Microsoft Windows) does something better.
>I don't have the English version of it, but when I try to create an 
>image of 20000 x 20000 pixels on my system (your system may be 
>different), it waits a couple of seconds, then displays a message 
>similar to "Insufficient memory or resources. Close one or more 
>applications and retry", and it does not terminate, keeping open the 
>previous bitmap. Obviously it tried to create such a large bitmap and 
>failed, because if it had decided that such a bitmap was too large, it 
>wouldn't wait that time, and it wouldn't suggest to close other 
>applications.

Well, this can be implemented relatively simply if, like MS-Paint, you
are willing to preallocate all the memory for a new bitmap.  On create
new bitmap request, simply try{ allocate memory for x*y pixels } and 
catch(std::bad_alloc &).  

Most of the time, the std::bad_alloc will be thrown because the
allocation attempt was too large so the error message should work.
However, it might happen that even attempting to print the error
message later will fail.   

[toc] | [prev] | [next] | [standalone]

#13338

On 9 Gen, 18:41, yatremblay@bel1lin202.(none) (Yannick Tremblay)
wrote:
> In article <4f00ee2d$0$6836$5fc3...@news.tiscali.it>,CarloMilanesi <carlo.nospam.milan...@libero.it> wrote:
>
> Most of the time, the std::bad_alloc will be thrown because the
> allocation attempt was too large so the error message should work.
> However, it might happen that even attempting to print the error
> message later will fail.

Usually, when a std::bad_alloc exception is raised, the unrolling of
the stack causes the deallocation of some other memory, and therefore,
even if the "new" causing the exception was asking for just a couple
of bytes, when the exception is catched probably enough memory is
available to display an error message.
Therefore most business or consumer application developers shouldn't
care about the possibility that the error handling procedure cannot
properly run beacuse of low memory conditions.
However, if there is such a concern, the error handling subsystem
should create at program startup all the objects that it will need.
--
Carlo Milanesi

[toc] | [prev] | [next] | [standalone]

#12944

On Sat, 2011-12-31, Victor Bazarov wrote:
> On 12/31/2011 8:56 AM, Datesfat Chicks wrote:
>> I'm just learning C++ (after being a 19-year C veteran), so forgive
>> any naive questions or concerns.
>>
>> It seems to me that C++ has more inherent reliance on dynamic memory
>> allocation than C.  It doesn't have to be that way in all cases (many
>> classes would be implemented without dynamic allocation), but it seems
>> more natural in C++ that dynamic allocation would appear in programs.
>
> The reality is simpler: it's easier to do dynamic memory properly in C++ 
> (purposefully so, actually), and that's the reason more people resort to 
> it.  In fact I've written plenty of code that never allocated anything 
> in the free store (except by any library mechanisms that I had no 
> control over).  You don't have to use dynamic memory if you don't need it.

Another angle on the same answer:

I tend to use much less explicit dynamic allocation in C++ than in C,
but that's because the standard containers like std::vector,
std::string and std::map replace malloc()ed arrays (and the nasty
fixed-size arrays).

I believe I do /more/ dynamic allocation in C++ -- if you count the
ones done for me by the standard containers.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#12917

On Dec 31, 1:56 pm, Datesfat Chicks <datesfat.chi...@gmail.com> wrote:

<snip>

> It seems to me that C++ has more inherent reliance on dynamic memory
> allocation than C.

I'm not sure this is true. Dynamic memory allocation is a problem
space property rather than a language property.

You need dynamic allocationwhen you can't predict in advance how many
objects of a particular type you're going to need (same as C). It's
also handy for polymorphism (maybe I'm contradicting myself here!).
And for inherently dynamic structures- like trees or linked lists
(same as C).

You also avoid doing memory management yourself by using std::vector
and other standard library containers. If you do use dynamic memory
make sure you wrap them up in auto_ptr or the newer unique_ptr and so
on.

> It doesn't have to be that way in all cases (many
> classes would be implemented without dynamic allocation), but it seems
> more natural in C++ that dynamic allocation would appear in programs.
>
> Here are my questions:
>
> a)I'm assuming that malloc() and free() are still available in C++?

yes, but its unusual to use them. The main reason would be to
interface with C code that needed malloc'dmemory.

> b)Do these draw from the same memory pool as "new" and "delete"?

no. An implementaion might but there's no guarantee. If you new it you
must delete it and if you malloc you must free.

> My reason for asking that question is that I'm looking for a graceful
> strategy if dynamic memory gets exhausted.

to be honest there is no graceful strategy. You could count how many
objects you've allocated and get panicy when a lot have been
allocated.

> The obvious strategy that comes to mind is:
>
> a)When the program starts, grab a good chunk of memory using malloc().

well I'd new a bunch of unsigned chars- which is moral equivalent of
malloc()

> b)If the dynamic memory gets exhausted via "new" and friends, handle
> the exception.  In the exception handler, free() the memory allocated
> by (a) and set a flag to signal the program to gracefully exit as soon
> as it can.

the problem is your compter is likely to crash in one of its other
processes. You better make sure any object you throw doesn't do
anything with dynamic memory.

> c)Hopefully the memory released by free() would be enough to allow the
> program to exit gracefully (rather than a sudden less controlled
> exit).
>
> Is this sane?  Is there a better way?

I've never done it but it seems workable. I usually assume I'm in such
deep shit when I get an allocation error I just shut things down. And
it usually turns out to be some terible error.

[toc] | [prev] | [next] | [standalone]

#12918

Datesfat Chicks <datesfat.chicks@gmail.com> schrieb:

> a)I'm assuming that malloc() and free() are still available in C++?

Yes, they are. And also other platform dependend functions, too
(realloc, calloc).

> b)Do these draw from the same memory pool as "new" and "delete"?

Yes, they do. Normally new and delete are just wrappers. They call
malloc and free internally. But there's no guarantee for this, they can
also use other strategies, but the pool is truely the same.

Note:

- It is not guaranteed that new and free or malloc and delete are
  compatible. Never mix this!

- There are two deletes: delete and delete[].

The best way to write clean, safe and portable C++ code is to FORGET the
existence of malloc and free (and all the other alloc-functions, strdup
and others) completely. NEVER USE these functions. You can do everything
(except reallocation) with new and delete.

> My reason for asking that question is that I'm looking for a graceful
> strategy if dynamic memory gets exhausted.
>
> The obvious strategy that comes to mind is:
>
> a)When the program starts, grab a good chunk of memory using malloc().
>
> b)If the dynamic memory gets exhausted via "new" and friends, handle
> the exception.  In the exception handler, free() the memory allocated
> by (a) and set a flag to signal the program to gracefully exit as soon
> as it can.
>
> c)Hopefully the memory released by free() would be enough to allow the
> program to exit gracefully (rather than a sudden less controlled
> exit).

There's no need for the initial malloc call. You can allocate the chunk
also by new and free it by delete.

But I would avoid this topic completely. I think the need to react on
memory exaustion is very few, especially if you just learn the language.
If this really happens something else is wrong.

A "memory management" case that is much more interesting (because it
really happens) is stack overflow. The stack overflow is amazing: you
will not be able to call any further function because there's just no
space left on the damned stack ...

T.M.

[toc] | [prev] | [next] | [standalone]

#12939

On Sat, 31 Dec 2011 08:56:29 -0500, Datesfat Chicks wrote:

> It seems to me that C++ has more inherent reliance on dynamic memory
> allocation than C.

The STL relies upon dynamic memory where libc normally requires the caller
provide buffers. The C++ language doesn't rely upon it.

> a)I'm assuming that malloc() and free() are still available in C++?

Yes.

> b)Do these draw from the same memory pool as "new" and "delete"?

It isn't specified, so you should assume that they don't.

> My reason for asking that question is that I'm looking for a graceful
> strategy if dynamic memory gets exhausted.
> 
> The obvious strategy that comes to mind is:
> 
> a)When the program starts, grab a good chunk of memory using malloc().
> 
> b)If the dynamic memory gets exhausted via "new" and friends, handle
> the exception.  In the exception handler, free() the memory allocated
> by (a) and set a flag to signal the program to gracefully exit as soon
> as it can.
> 
> c)Hopefully the memory released by free() would be enough to allow the
> program to exit gracefully (rather than a sudden less controlled
> exit).
> 
> Is this sane?  Is there a better way?

If you're programming in C++, a trivial modification to your strategy
would be to grab a chunk of memory using new[] rather than malloc() and
release it using delete[] rather than free().

But you're probably better off designing your "graceful failure"
strategy so that it doesn't depend upon additional memory, rather than
trying to guess how much memory you need to reserve for it. Certainly,
that's likely to be harder to do in C++ than in C (and even in C, it's
easier if you use the OS' native APIs rather than the portable C APIs).

One final point: on Linux, using too much memory often results not in a
std::bad_alloc exception (or malloc() returning NULL) but in SIGKILL from
the kernel's "OOM killer", in which case any recovery strategy is doomed.
Even without that aspect, you can't control how third-party libraries
handle an out-of-memory situation; they may simply abort(), or not bother
checking the value returned from malloc() and promptly segfault upon
dereferencing the NULL pointer.

[toc] | [prev] | [next] | [standalone]

#12999

On Sun, 01 Jan 2012 09:16:38 +0000, Nobody <nobody@nowhere.com> wrote:
>
>One final point: on Linux, using too much memory often results not in a
>std::bad_alloc exception (or malloc() returning NULL) but in SIGKILL from
>the kernel's "OOM killer", in which case any recovery strategy is doomed.
>Even without that aspect, you can't control how third-party libraries
>handle an out-of-memory situation; they may simply abort(), or not bother
>checking the value returned from malloc() and promptly segfault upon
>dereferencing the NULL pointer.

Both good points.  I had made the assumption that it would be MY code
running out of memory.

Still, if the exception is handled and if only "new" is used, the
approach I suggested should still work.  Assuming (now that I think
about it) that the exception handler is able to cause the failed new
call to succeed, which might not be possible (I need to look up how
this exception is done and whether new gets a second chance).

DFC

[toc] | [prev] | [next] | [standalone]

#13023

On Jan 2, 10:48 pm, Datesfat Chicks <datesfat.chi...@gmail.com> wrote:
> On Sun, 01 Jan 2012 09:16:38 +0000, Nobody <nob...@nowhere.com> wrote:
>
> >One final point: on Linux, using too much memory often results not in a
> >std::bad_alloc exception (or malloc() returning NULL) but in SIGKILL from
> >the kernel's "OOM killer", in which case any recovery strategy is doomed.
> >Even without that aspect, you can't control how third-party libraries
> >handle an out-of-memory situation; they may simply abort(), or not bother
> >checking the value returned from malloc() and promptly segfault upon
> >dereferencing the NULL pointer.
>
> Both good points.  I had made the assumption that it would be MY code
> running out of memory.

Actually, these are wrong points in this newsgroup. In C and C++,
there's no such thing as OOM killer, there's no SIGKILL. There's no
such thing on other systems either, and even on systems who do use
overcommit, it's an optional feature. Not bothering with checking can
be a strategy on OOM-enabled systems, sure, but is not a good general
answer, and is certainly not meaningful in the light of standard C or C
++. That said, proper C++ code will not "check" anything anyhow; it
will possibly catch an exception later, and upon a stack unwind.

As for libraries who abort() upon OOM (or segfault), I think that
everyone would agree that they have a design bug. I mean, what they
are doing is akin to coming into someone's house and shitting on the
floor because someone else was in the bathroom.

Goran.

[toc] | [prev] | [next] | [standalone]

#13031

On Tue, 3 Jan 2012 00:02:21 -0800 (PST), Goran <goran.pusic@gmail.com>
wrote:

>On Jan 2, 10:48 pm, Datesfat Chicks <datesfat.chi...@gmail.com> wrote:
>> On Sun, 01 Jan 2012 09:16:38 +0000, Nobody <nob...@nowhere.com> wrote:
>>
>> >One final point: on Linux, using too much memory often results not in a
>> >std::bad_alloc exception (or malloc() returning NULL) but in SIGKILL from
>> >the kernel's "OOM killer", in which case any recovery strategy is doomed.
>> >Even without that aspect, you can't control how third-party libraries
>> >handle an out-of-memory situation; they may simply abort(), or not bother
>> >checking the value returned from malloc() and promptly segfault upon
>> >dereferencing the NULL pointer.
>>
>> Both good points.  I had made the assumption that it would be MY code
>> running out of memory.
>
>Actually, these are wrong points in this newsgroup. In C and C++,
>there's no such thing as OOM killer, there's no SIGKILL. There's no
>such thing on other systems either, and even on systems who do use
>overcommit, it's an optional feature. Not bothering with checking can
>be a strategy on OOM-enabled systems, sure, but is not a good general
>answer, and is certainly not meaningful in the light of standard C or C
>++. That said, proper C++ code will not "check" anything anyhow; it
>will possibly catch an exception later, and upon a stack unwind.

I'm a little surprised at the SIGKILL response.  If you're going to
have a std::bad_alloc exception, then it seems an improper
implementation if that exception is useless because the process gets
killed by another mechanism before the exception is thrown.

>As for libraries who abort() upon OOM (or segfault), I think that
>everyone would agree that they have a design bug. I mean, what they
>are doing is akin to coming into someone's house and shitting on the
>floor because someone else was in the bathroom.

I do that even if someone else is in the bathroom.

However, for people I really like, I put a little plastic wrap or
aluminum foil down and do it on that.  That way you get the aroma and
shock effect without the cleanup difficulties.

DFC

[toc] | [prev] | [next] | [standalone]

#13040

On Tue, 03 Jan 2012 12:53:47 -0500, Datesfat Chicks wrote:

> I'm a little surprised at the SIGKILL response.  If you're going to
> have a std::bad_alloc exception, then it seems an improper
> implementation if that exception is useless because the process gets
> killed by another mechanism before the exception is thrown.

They serve different purposes. std::bad_alloc is thrown if you want more
memory but it cannot be provided (i.e. it isn't available from the heap
and the OS won't let you enlarge the heap). SIGKILL occurs if the OS
decides that it wants the memory which your process already has.

The latter is primarily an issue on systems where overcommit is enabled.
Overcommit being enabled is the norm; if overcommit is disabled, the OS
won't allow a process to fork() unless it can provide sufficient virtual
memory to back all writeable pages in the process' address space. This is
usually overkill, as most process which call fork() will promptly call
execve() having modified only a tiny fraction of their writable memory.

The problem comes when the kernel finds that it actually needs more memory
than is available, e.g. because one copy-on-write page too many has
actually been written to. The only way out of that situation is to revoke
allocations which have already been granted by sacrificing an existing
process. When doing so, it will typically choose the most "greedy"
unprivileged process available. Any process where handling out-of-memory
conditions was a significant design consideration would seem to be a
likely candidate to be sacrificed.

BTW: regarding your original problem (how to handle the C++ side of
allocation failures), I remembered std::set_new_handler(). If a handler
has been registered via this function, operator new should[1] call it if
it cannot obtain the desired memory. In fact, it should call it repeatedly
until either the allocation succeeds or the handler unregisters itself
(IOW: if you write such a handler, it must unregister itself if it cannot
release any more memory, otherwise operator new will get stuck in an
infinite loop). std::bad_alloc will only be thrown if allocation fails and
no handler is registered.

[1] The global operator new can be relied upon to do so; user-defined
versions of operator new may be a different matter.

[toc] | [prev] | [next] | [standalone]

#13069

On Jan 3, 6:53 pm, Datesfat Chicks <datesfat.chi...@gmail.com> wrote:
> On Tue, 3 Jan 2012 00:02:21 -0800 (PST), Goran <goran.pu...@gmail.com>
> wrote:
>
>
>
>
>
>
>
>
>
> >On Jan 2, 10:48 pm, Datesfat Chicks <datesfat.chi...@gmail.com> wrote:
> >> On Sun, 01 Jan 2012 09:16:38 +0000, Nobody <nob...@nowhere.com> wrote:
>
> >> >One final point: on Linux, using too much memory often results not in a
> >> >std::bad_alloc exception (or malloc() returning NULL) but in SIGKILL from
> >> >the kernel's "OOM killer", in which case any recovery strategy is doomed.
> >> >Even without that aspect, you can't control how third-party libraries
> >> >handle an out-of-memory situation; they may simply abort(), or not bother
> >> >checking the value returned from malloc() and promptly segfault upon
> >> >dereferencing the NULL pointer.
>
> >> Both good points. I had made the assumption that it would be MY code
> >> running out of memory.
>
> >Actually, these are wrong points in this newsgroup. In C and C++,
> >there's no such thing as OOM killer, there's no SIGKILL. There's no
> >such thing on other systems either, and even on systems who do use
> >overcommit, it's an optional feature. Not bothering with checking can
> >be a strategy on OOM-enabled systems, sure, but is not a good general
> >answer, and is certainly not meaningful in the light of standard C or C
> >++. That said, proper C++ code will not "check" anything anyhow; it
> >will possibly catch an exception later, and upon a stack unwind.
>
> I'm a little surprised at the SIGKILL response.  If you're going to
> have a std::bad_alloc exception, then it seems an improper
> implementation if that exception is useless because the process gets
> killed by another mechanism before the exception is thrown.

Well... malloc returns NULL if allocation request can't be fulfilled,
or a pointer pointing to a block of requested size. That's standard C.
In standard C++, new returns the object (objects), or throws
bad_alloc. This is standard C++. This is the scope of this group.

Now... On systems with OOM killer, malloc might return a pointer, and
code would die when trying to use it (likely, to write to it). Similar
to new, but there, if OOM killer is to kick in, it is slightly more
probable that OOM killer will kick in __sooner__, because in C++, it
is more probable that construction will happen right after the
allocation.

When does OOM killer kick in, in practice? When there is address space
for the request, but system over-comitted when it granted that memory,
then ran out of virtual memory and wants to get rid of the process it
found offensive. So even on OOM-killer enabled systems, bad_alloc is
not useless - it still covers the "no address space, buddy" case.

Finally, what is well-behaved code to do about OOM killer on such
systems? IMHO, have a process-restart strategy, and data-loss-
prevention strategy. Indeed, such systems do provide restart
mechanisms.

> >As for libraries who abort() upon OOM (or segfault), I think that
> >everyone would agree that they have a design bug. I mean, what they
> >are doing is akin to coming into someone's house and shitting on the
> >floor because someone else was in the bathroom.
>
> I do that even if someone else is in the bathroom.
>
> However, for people I really like, I put a little plastic wrap or
> aluminum foil down and do it on that.  That way you get the aroma and
> shock effect without the cleanup difficulties.

I guess you fund yourself offended by my analogy because your own code
aborts or segfaults on NULL from malloc. There's no need for that.
Nobody cares if you do that, if they don't have to deal with such
code. And if code only runs on a system with OOM killer... I guess it
has to be said that this strategy is arguably more such-system-
friendly than standard C or C++. But this is very much out of scope
for this newsgroup.

Goran.

[toc] | [prev] | [next] | [standalone]

#13070

On Wed, 04 Jan 2012 00:11:41 -0800, Goran wrote:

> So even on OOM-killer enabled systems, bad_alloc is
> not useless - it still covers the "no address space, buddy" case.

It also covers the case where the process is subject to explicit resource
limits which are set well below the total amount of virtual memory on the
system.

Even on systems which lack an OOM-killer, memory-hungry processes still
have to worry about issues other than user-space allocation failure
(std::bad_alloc or a NULL return from malloc() or nothrow-new). E.g. if
the system has "excessive" amounts of swap, allocation will succeed and
the process won't get OOM-killed, but it may run so slowly that just
having it killed would have been preferable.

IOW, the "official" allocation-failure mechanism is only one piece of the
puzzle, and it's only worth solving if you can also deal with the other
aspects.

[toc] | [prev] | [next] | [standalone]

#13086

On Sun, 2012-01-01, Nobody wrote:
...
> One final point: on Linux, using too much memory often results not in a
> std::bad_alloc exception (or malloc() returning NULL) but in SIGKILL from
> the kernel's "OOM killer", [...]

I believe you (and the followups) are using the word "OOM killer"
incorrectly here.  The "OOM killer" of Linux is/was a feature which
kills random processes when the OS is desperate for memory. Those
processes don't have to allocate any memory at all in order to be
candidates for killing.

  http://linux-mm.org/OOM_Killer

Possibly, the word you want is "overcommitting".

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#13097

On Wed, 04 Jan 2012 19:28:51 +0000, Jorgen Grahn wrote:

>> One final point: on Linux, using too much memory often results not in a
>> std::bad_alloc exception (or malloc() returning NULL) but in SIGKILL from
>> the kernel's "OOM killer", [...]
> 
> I believe you (and the followups) are using the word "OOM killer"
> incorrectly here.  The "OOM killer" of Linux is/was a feature which
> kills random processes when the OS is desperate for memory.

Yes, I'm aware of this, and thought that I had explained it.

> Those processes don't have to allocate any memory at all in order to be
> candidates for killing.
> 
>   http://linux-mm.org/OOM_Killer

That link describes the factors which are used to choose a process to
sacrifice. One of them is the amount of memory which it is using. In fact,
it says:

> 3) we don't kill anything innocent of eating tons of memory

That memory doesn't have to come from malloc/new (it could be a result of
modifying large portions of the data and/or bss segments, or using a lot
of stack), but a process which doesn't use much memory one way or another
isn't a likely candidate for being sacrificed.

> Possibly, the word you want is "overcommitting".

Overcommit is the reason the kernel needs an OOM-killer. If you disable
overcommit, the kernel will never find itself in the position of needing
to sacrifice a process in order to dig itself out of the hole that
overcommitting got it into. But if you do that, don't expect large
processes (e.g. Firefox) to be able to spawn any child processes; the
fork() will fail unless there's enough free memory to duplicate the
parent's entire address space, even though 99% of it will be discarded as
soon as exec() is called.

[toc] | [prev] | [next] | [standalone]

#13118

On Wed, 2012-01-04, Nobody wrote:
> On Wed, 04 Jan 2012 19:28:51 +0000, Jorgen Grahn wrote:
>
>>> One final point: on Linux, using too much memory often results not in a
>>> std::bad_alloc exception (or malloc() returning NULL) but in SIGKILL from
>>> the kernel's "OOM killer", [...]
>> 
>> I believe you (and the followups) are using the word "OOM killer"
>> incorrectly here.  The "OOM killer" of Linux is/was a feature which
>> kills random processes when the OS is desperate for memory.
>
> Yes, I'm aware of this, and thought that I had explained it.

You might have -- I was reading snipped followups in some subthread when
I noticed what I believed was odd terminology, and didn't read all the
rest.

>> Possibly, the word you want is "overcommitting".
>
> Overcommit is the reason the kernel needs an OOM-killer.

OK, I wasn't aware of that.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#12943

Am 31.12.11 14:56, schrieb Datesfat Chicks:
> My reason for asking that question is that I'm looking for a graceful
> strategy if dynamic memory gets exhausted.

Depending on what your program does, there might be another strategy. If 
it is doing a complicated computation, lets say an FEM solver, then you 
can run the computational kernel in another process. The GUI process 
would communicate using stdio or TCP/IP with the computation to show 
progress and so on. When the computation process runs aout of memory, it 
is simply killed by the OS, which does not hurt because its resources 
like files, memory, network connections are then freed automatically by 
the OS. The GUI would just notice that the process is killed and inform 
the user, write log files and so on.

	Christian

[toc] | [prev] | [next] | [standalone]

#12947

"Christian Gollwitzer" <auriocus@gmx.de> ha scritto nel messaggio 
news:jdq47d$5fp$1@dont-email.me...
> Am 31.12.11 14:56, schrieb Datesfat Chicks:
>> My reason for asking that question is that I'm looking for a graceful
>> strategy if dynamic memory gets exhausted.
>
> Depending on what your program does, there might be another strategy. If it is 
> doing a complicated computation, lets say an FEM solver, then you can run the 
> computational kernel in another process. The GUI process would communicate 
> using stdio or TCP/IP with the computation to show progress and so on. When 
> the computation process runs aout of memory, it is simply killed by the OS, 
> which does not hurt because its resources like files, memory, network 
> connections are then freed automatically by the OS. The GUI would just notice 
> that the process is killed and inform the user, write log files and so on.
>
> Christian

don't know whou you are and what you use to program
but is the programmer that has to have the controcll,
not the OS nor the sys

is possible if one malloc fail, someone
can clear and free all without request a more singel bit to the sys
 

[toc] | [prev] | [next] | [standalone]

#13000

On Sun, 01 Jan 2012 18:13:20 +0100, Christian Gollwitzer
<auriocus@gmx.de> wrote:

>Am 31.12.11 14:56, schrieb Datesfat Chicks:
>> My reason for asking that question is that I'm looking for a graceful
>> strategy if dynamic memory gets exhausted.
>
>Depending on what your program does, there might be another strategy. If 
>it is doing a complicated computation, lets say an FEM solver, then you 
>can run the computational kernel in another process. The GUI process 
>would communicate using stdio or TCP/IP with the computation to show 
>progress and so on. When the computation process runs aout of memory, it 
>is simply killed by the OS, which does not hurt because its resources 
>like files, memory, network connections are then freed automatically by 
>the OS. The GUI would just notice that the process is killed and inform 
>the user, write log files and so on.

Yours may be the most robust solution suggested, and it has other
advantages as well, such as:

a)Separates the GUI from the computation.

b)Might be more friendly to multiprocessor systems (especially if the
computation involves more than one process).

c)Facillitates an approach that might scale well to clusters.

Good call.

DFC
DFC

[toc] | [prev] | [next] | [standalone]

#13114

Datesfat Chicks <datesfat.chicks@gmail.com> writes:

> .. c)Hopefully the memory released by free() would be enough to allow the
> program to exit gracefully (rather than a sudden less controlled
> exit).
>
> Is this sane?  Is there a better way?

That sounds fine in principle, I think. 

I'd expect malloc() & free() to allocate from the same heap as new &
delete. But if you were concerned, you could always new -- rather than
malloc() -- a large array at start-up & then delete that once the memory
got low.

Scott Meyers has something to say about this in "Effective C++" -- I
have edition-II, in which this appears in Item 7: Be prepared for
out-of-memory conditions.

-- s.

[toc] | [prev] | [next] | [standalone]

#13216

On Saturday, December 31, 2011 8:56:29 AM UTC-5, Datesfat Chicks wrote:
> I'm just learning C++ (after being a 19-year C veteran), so forgive
> any naive questions or concerns.
> 
> It seems to me that C++ has more inherent reliance on dynamic memory
> allocation than C.  It doesn't have to be that way in all cases (many
> classes would be implemented without dynamic allocation), but it seems
> more natural in C++ that dynamic allocation would appear in programs.

No it doesn't.  In fact, each new version of C++ expands other forms of programming outside the "dynamically allocate OOP objects and dispatch messages" model.

However, the big advantage C++ has over C is encapsulation and modeling.  What you had to do in C with manually with "gentleman's agreements" you can now have C++ handle automatically.  For example, the latest version of C++ (2011) lets you specify with built-in integer type will be used as an enum's base type, something you had to do before with a gentleman's agreement via adding a sentinel value of MAX_LONG (or whatever).  Also, this change allows base types smaller than int.

Part of the standard library is several kinds of containers, which encapsulate memory calls for you.

> 
> Here are my questions:
> 
> a)I'm assuming that malloc() and free() are still available in C++?

Yes, all of the C library is accessible from C++.  (C-1999 for C++-2011, C-1989 for previous C++ versions.)  C++ will describe some C functions with added "const" and/or removed "restrict" qualifications.

> 
> b)Do these draw from the same memory pool as "new" and "delete"?

They may or may not.  The only restriction (AFAIK) is that malloc/free cannot secretly use new/delete.  (I guess because this lets the reverse happen without infinite recursion.)  You must use the allocator's designated counter function to do deallocations; no mix-and-match.

> 
> My reason for asking that question is that I'm looking for a graceful
> strategy if dynamic memory gets exhausted.
> 
> The obvious strategy that comes to mind is:
> 
> a)When the program starts, grab a good chunk of memory using malloc().
> 
> b)If the dynamic memory gets exhausted via "new" and friends, handle
> the exception.  In the exception handler, free() the memory allocated
> by (a) and set a flag to signal the program to gracefully exit as soon
> as it can.
> 
> c)Hopefully the memory released by free() would be enough to allow the
> program to exit gracefully (rather than a sudden less controlled
> exit).
> 
> Is this sane?  Is there a better way?

I don't think it's sane.  Think about it: any reserve block large enough to make a difference would be big enough to contribute to the initial problem!  And what if the deallocation routine doesn't marks the memory as free-for-all, but holds on to it a little longer in case of another allocation?  (This may not be a problem depending on which allocation routine is used next.)

I would say just let the memory crunch happen.  If you design your objects right, only the ones directly affected by the lack of memory should be broken down, leaving the others as-is.

Daryle W.

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | comp.lang.c++

csiph-web