Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #26252 > unrolled thread

Failing Malloc

Back to article view | Back to comp.lang.c

Contents

  Failing Malloc Eric <nospam@nospam.com> - 2012-09-12 20:07 +0000
    Re: Failing Malloc Xavier Roche <xroche@free.fr.NOSPAM.invalid> - 2012-09-12 22:12 +0200
      Re: Failing Malloc William Ahern <william@wilbur.25thandClement.com> - 2012-09-12 14:25 -0700
    Re: Failing Malloc Andrew Cooper <andrew@nospam.example.com> - 2012-09-12 21:21 +0100
      Re: Failing Malloc Eric <nospam@nospam.com> - 2012-09-12 20:34 +0000
        Re: Failing Malloc Paul N <gw7rib@aol.com> - 2012-09-12 13:46 -0700
          Re: Failing Malloc Eric <nospam@nospam.com> - 2012-09-13 20:21 +0000
            Re: Failing Malloc Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-09-13 17:31 -0400
            Re: Failing Malloc James Kuyper <jameskuyper@verizon.net> - 2012-09-13 17:57 -0400
        Re: Failing Malloc John Gordon <gordon@panix.com> - 2012-09-12 20:55 +0000
          Re: Failing Malloc William Ahern <william@wilbur.25thandClement.com> - 2012-09-12 14:33 -0700
          Re: Failing Malloc falk@rahul.net (Edward A. Falk) - 2012-09-13 22:43 +0000
        Re: Failing Malloc James Kuyper <jameskuyper@verizon.net> - 2012-09-12 17:37 -0400
        Re: Failing Malloc Johann Klammer <klammerj@NOSPAM.a1.net> - 2012-09-13 08:57 +0200
    Re: Failing Malloc James Kuyper <jameskuyper@verizon.net> - 2012-09-12 17:20 -0400
      Re: Failing Malloc Ian Collins <ian-news@hotmail.com> - 2012-09-13 09:27 +1200
        Re: Failing Malloc James Kuyper <jameskuyper@verizon.net> - 2012-09-12 17:56 -0400
          Re: Failing Malloc Ian Collins <ian-news@hotmail.com> - 2012-09-13 10:28 +1200
            Re: Failing Malloc James Kuyper <jameskuyper@verizon.net> - 2012-09-12 18:57 -0400
              Re: Failing Malloc William Ahern <william@wilbur.25thandClement.com> - 2012-09-12 16:21 -0700
              Re: Failing Malloc Ian Collins <ian-news@hotmail.com> - 2012-09-13 12:07 +1200
                Re: Failing Malloc James Kuyper <jameskuyper@verizon.net> - 2012-09-13 13:18 -0400
                  Re: Failing Malloc Ian Collins <ian-news@hotmail.com> - 2012-09-14 07:37 +1200
                    Re: Failing Malloc James Kuyper <jameskuyper@verizon.net> - 2012-09-13 16:05 -0400
                      Re: Failing Malloc William Ahern <william@wilbur.25thandClement.com> - 2012-09-13 14:21 -0700
                        Re: Failing Malloc James Kuyper <jameskuyper@verizon.net> - 2012-09-13 17:38 -0400
                    Re: Failing Malloc William Ahern <william@wilbur.25thandClement.com> - 2012-09-13 14:03 -0700
                      Re: Failing Malloc Ian Collins <ian-news@hotmail.com> - 2012-09-15 10:01 +1200
              Re: Failing Malloc Philip Lantz <prl@canterey.us> - 2012-09-14 02:50 -0700
    Re: Failing Malloc Kaz Kylheku <kaz@kylheku.com> - 2012-09-12 21:36 +0000
    Re: Failing Malloc Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-09-12 17:41 -0400
    Re: Failing Malloc Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2012-09-12 16:07 -0600
      Re: Failing Malloc "christian.bau" <christian.bau@cbau.wanadoo.co.uk> - 2012-09-12 16:03 -0700
    Re: Failing Malloc Kenneth Brody <kenbrody@spamcop.net> - 2012-09-12 19:24 -0400
    Re: Failing Malloc The Great Firewall of China Blue <chine.bleu@yahoo.com> - 2012-09-12 16:34 -0700
    Re: Failing Malloc Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-09-13 00:14 -0700

Page 2 of 2 — ← Prev page 1 [2]

#26283

On 09/13/12 10:57 AM, James Kuyper wrote:
> On 09/12/2012 06:28 PM, Ian Collins wrote:
> ....
>> I understand your pain.  I use a similar approach to resource allocation
>> failures to yours in C++ with exceptions that convey file and line
>> number information from the call site.  These can ether propagate up or
>> be caught and augmented and re-thrown higher up the stack.  I guess I
>> haven't done anything similar in C because it takes more work and lot
>> more discipline!
>
> On my new task, I'm actually using C++, and I've been trying to figure
> out how to make use of exceptions to produce the same kind of error
> output I get from the my C programs. It's a fantasy, since the new group
> I'm working with has it's own well-entrenched error handling conventions
> that would not mix well with my preferred style.
>
> However, when I did indulge that fantasy, I found that one key problem
> was that exceptions discard information about where they came from, as
> the call stack unwinds. Unless I catch and rethrow around every function
> call that could throw an exception (including the implicit function
> calls due to ctors, dtors, and operator overloads), it can't duplicate
> the information generated by my error messages.

I use the OS provided stack trace functions to build a stack trace to 
include in the exception, so there's no need to pick anything use up 
along the way.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#26304

On 09/12/2012 08:07 PM, Ian Collins wrote:
...
> I use the OS provided stack trace functions to build a stack trace to 
> include in the exception, so there's no need to pick anything use up 
> along the way.

My code is allowed to assume support for any feature mandated by POSIX
1003.2-1992 (except for a long list of prohibited functions - but none
of those seem relevant), but is not otherwise allowed to depend upon
OS-specific features. Do you know if any such functions were included in
that version of POSIX?

If not, can you at least identify the OS you're using, and the specific
stack trace functions you're talking about for that OS? I'd like to know
how they work - off-hand, functionality that would be an adequate
substitute for what I'm currently doing seems implausible.

[toc] | [prev] | [next] | [standalone]

#26307

On 09/14/12 05:18 AM, James Kuyper wrote:
> On 09/12/2012 08:07 PM, Ian Collins wrote:
> ....
>> I use the OS provided stack trace functions to build a stack trace to
>> include in the exception, so there's no need to pick anything use up
>> along the way.
>
> My code is allowed to assume support for any feature mandated by POSIX
> 1003.2-1992 (except for a long list of prohibited functions - but none
> of those seem relevant), but is not otherwise allowed to depend upon
> OS-specific features. Do you know if any such functions were included in
> that version of POSIX?
>
> If not, can you at least identify the OS you're using, and the specific
> stack trace functions you're talking about for that OS? I'd like to know
> how they work - off-hand, functionality that would be an adequate
> substitute for what I'm currently doing seems implausible.

I think most of this stuff is OS specific and there's a big whole in 
POSIX where it should be.

I mainly use Solaris which has a family of stack reading functions:

http://docs.oracle.com/cd/E23824_01/html/821-1465/walkcontext-3c.html#scrolltoc

Googling for "solaris walkcontext" will find the Linux near equivalents.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#26309

On 09/13/2012 03:37 PM, Ian Collins wrote:
...
> I think most of this stuff is OS specific and there's a big whole in 

"hole"?

> POSIX where it should be.
> 
> I mainly use Solaris which has a family of stack reading functions:
> 
> http://docs.oracle.com/cd/E23824_01/html/821-1465/walkcontext-3c.html#scrolltoc
> 
> Googling for "solaris walkcontext" will find the Linux near equivalents.

Those functions seem to be missing something, that I expected to be
missing, that would be necessary in order to produce output containing
information comparable to that produced by my current error messages. I
compared that information to an "annotated stack trace". The "annotated"
is important: it refers to the fact that for every function in the call
stack, the error messages include the values of all scalar non-pointer
parameters, and the text of any strings pointed at by character pointer
input parameters.

Less consistently, if an input pointer parameter points at a struct
containing a single member that identifies what the struct contents
represent, the message will sometimes include the value of that member
(or of the string it points at). Example: one of the packages I use
provides a file handle structure that allows me to print out
file_handle->filename.

gdb can provide similar information when debugging a program compiled
with -g, but when I followed that URL, it was not clear to me that the
walkcontext routines have any features that would enable such annotation.

Another aspect of the "annotation" is that the error messages also
contain the value of the loop index for any loop containing the code
that generates the error message (or, more usefully, a string associated
with that index value, if one is available), but that's functionality I
could not reasonably expect a stack-walking routine to provide.

[toc] | [prev] | [next] | [standalone]

#26317

James Kuyper <jameskuyper@verizon.net> wrote:
<snip>
> Another aspect of the "annotation" is that the error messages also
> contain the value of the loop index for any loop containing the code
> that generates the error message (or, more usefully, a string associated
> with that index value, if one is available), but that's functionality I
> could not reasonably expect a stack-walking routine to provide.

Perhaps not from one of the billion generic stack walking libraries out
there which all do the same thing (mostly useless, as most of the time the
execution paths to any particular point in application code can usually be
counted on a single hand missing 4 fingers). But it's not that difficult to
get more relevant information. For example, DWARF symbols provide the
ability to introspect every object on your stack. Crudely speaking, you can
write a library which takes an address, and which returns a structure
describing the object most likely at that position. For example, for a
pointer to an automatic object you first derive the frame address, and then
you derive the address of the running function. You then use that
information to find the data structure describing the automatic variables in
play, including their identifers in the original source, which allow you to
actually name the object for which you have a reference.

Or you could just dump the stack frames, to be decoded later.

[toc] | [prev] | [next] | [standalone]

#26319

On 09/13/2012 05:21 PM, William Ahern wrote:
...
> ... most of the time the
> execution paths to any particular point in application code can usually be
> counted on a single hand missing 4 fingers).

Not if it's a utility routine that's called from  many different
locations in the code. In the program I'm working on right now, most of
the code seems to be in routines like that. Only routines from the top
two levels of the program are called from unique locations.

[toc] | [prev] | [next] | [standalone]

#26315

Ian Collins <ian-news@hotmail.com> wrote:
> On 09/14/12 05:18 AM, James Kuyper wrote:
> > On 09/12/2012 08:07 PM, Ian Collins wrote:
> > ....
> >> I use the OS provided stack trace functions to build a stack trace to
> >> include in the exception, so there's no need to pick anything use up
> >> along the way.
> >
> > My code is allowed to assume support for any feature mandated by POSIX
> > 1003.2-1992 (except for a long list of prohibited functions - but none
> > of those seem relevant), but is not otherwise allowed to depend upon
> > OS-specific features. Do you know if any such functions were included in
> > that version of POSIX?
> >
> > If not, can you at least identify the OS you're using, and the specific
> > stack trace functions you're talking about for that OS? I'd like to know
> > how they work - off-hand, functionality that would be an adequate
> > substitute for what I'm currently doing seems implausible.

> I think most of this stuff is OS specific and there's a big whole in 
> POSIX where it should be.

> I mainly use Solaris which has a family of stack reading functions:

> http://docs.oracle.com/cd/E23824_01/html/821-1465/walkcontext-3c.html#scrolltoc

> Googling for "solaris walkcontext" will find the Linux near equivalents.

On Linux there are two easy ways to get a stack trace:

1) glibc's backtrace() from <execinfo.h>.
2) GCC's/clang's __builtin_frame_address and __builtin_return_address.

glibc also has backtrace_symbols(), but it uses dladdr. The ELF and DWARF
symbols can get you more accurate information, more reliably. But reading
the ELF and DWARF bits directly is complex, thus libbfd--which like
__builtin_frame_address should actually work everywhere GCC/GDB exists,
including Solaris/SPARC.

[toc] | [prev] | [next] | [standalone]

#26379

On 09/14/12 09:03 AM, William Ahern wrote:
> Ian Collins<ian-news@hotmail.com>  wrote:
>> On 09/14/12 05:18 AM, James Kuyper wrote:
>>> On 09/12/2012 08:07 PM, Ian Collins wrote:
>>> ....
>>>> I use the OS provided stack trace functions to build a stack trace to
>>>> include in the exception, so there's no need to pick anything use up
>>>> along the way.
>>>
>>> My code is allowed to assume support for any feature mandated by POSIX
>>> 1003.2-1992 (except for a long list of prohibited functions - but none
>>> of those seem relevant), but is not otherwise allowed to depend upon
>>> OS-specific features. Do you know if any such functions were included in
>>> that version of POSIX?
>>>
>>> If not, can you at least identify the OS you're using, and the specific
>>> stack trace functions you're talking about for that OS? I'd like to know
>>> how they work - off-hand, functionality that would be an adequate
>>> substitute for what I'm currently doing seems implausible.
>
>> I think most of this stuff is OS specific and there's a big whole in
>> POSIX where it should be.
>
>> I mainly use Solaris which has a family of stack reading functions:
>
>> http://docs.oracle.com/cd/E23824_01/html/821-1465/walkcontext-3c.html#scrolltoc
>
>> Googling for "solaris walkcontext" will find the Linux near equivalents.
>
> On Linux there are two easy ways to get a stack trace:
>
> 1) glibc's backtrace() from<execinfo.h>.

That's good to know, Solaris has the same header/interface.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#26335

James Kuyper wrote:
> 
> On 09/12/2012 06:28 PM, Ian Collins wrote:
> ...
> > I understand your pain.  I use a similar approach to resource allocation 
> > failures to yours in C++ with exceptions that convey file and line 
> > number information from the call site.  These can ether propagate up or 
> > be caught and augmented and re-thrown higher up the stack.  I guess I 
> > haven't done anything similar in C because it takes more work and lot 
> > more discipline!
> 
> On my new task, I'm actually using C++, and I've been trying to figure
> out how to make use of exceptions to produce the same kind of error
> output I get from the my C programs. It's a fantasy, since the new group
> I'm working with has it's own well-entrenched error handling conventions
> that would not mix well with my preferred style.
> 
> However, when I did indulge that fantasy, I found that one key problem
> was that exceptions discard information about where they came from, as
> the call stack unwinds. Unless I catch and rethrow around every function
> call that could throw an exception (including the implicit function
> calls due to ctors, dtors, and operator overloads), it can't duplicate
> the information generated by my error messages.
> 
> If I did catch and rethrow everywhere, it would end up looking very
> similar to my current C code, which constantly checks for error codes.
> More than one person has looked at my code and come away with the
> impression that its primary purpose was error handling. One person even
> submitted a test plan for a module that triggered every single error
> message that could be generated by that module, but failed to cover
> normal processing.
> 
> I'd hoped to find a way to use C++ exceptions to reduce the error
> message clutter, but I can't figure out how to do so without a
> corresponding reduction in the amount of information conveyed by the
> error messages.


Perhaps you could define a class something like this:

class backtrace_handler
{
    // members that hold the information that would be
    // useful in a backtrace
    unsigned uns[4];
    char *ptr[4];
    double dbl[4];

    // a member that indicates which fields were initialized
    unsigned valid;
    
public:
    backtrace_handler(int a)
        : valid(0x0001)
        { uns[0] = a; }
    backtrace_handler(char *a)
        : valid(0x0002)
        { ptr[0] = a; }
    backtrace_handler(double a)
        : valid(0x0003)
        { dbl[0] = a; }
    backtrace_handler(char *a, double b)
        : valid(0x0032)
        { ptr[0] = a; dbl[1] = b; }
    backtrace_handler(double a, int b, char *c)
        : valid(0x0213)
        { dbl[0] = a; uns[1] = b; ptr[2] = c; }
    <additional constructors for different number/types of useful info>

    // Here's the key part:
    ~backtrace_handler()
    {
        //dump the members that were initialized
        int i = 0;
        while (valid != 0) {
            switch (valid & 0xf)
            {
                case 1: printf(..., uns[i]); break;
                case 2: printf(..., ptr[i]); break;
                ...
            }
            valid >>= 4;
            i++;
        }
    }
};

At the beginning of each function, declare an instance:

void f(int a, char *b)
{
    const backtrace_handler bth(a, b);

    ...
}

Obviously you would want to define the members and valid bits more 
cleanly, but I hope you get the idea. The destructor could be as complex 
as you need, to get the pretty output you want, but the constructors 
should be as simple as possible.

The obvious downside is that there is some cost to initialize the 
structure in each function. However, from your description, it seems 
that that may be minimal; it will essentially be simply spilling some of 
the function parameters to the stack, plus a single constant indicating 
which members were initialized and the order you want them displayed in 
the backtrace. (If the compiler's really smart, it might even be able to 
reuse the parameters that were spilled to the backtrace object instead 
of spilling them separately if there is register pressure. I'm not sure 
I've ever seen a compiler that was that smart, but I have seen some do 
some pleasantly surprising things.)

Oh, I guess you need to add the function name, too.

Unfortunately, I can't think of a way to get the line number within the 
function where the failed call occurred. But it's almost 3am; maybe I'll 
think of something tomorrow. :-)

[toc] | [prev] | [next] | [standalone]

#26265

On 2012-09-12, Eric <nospam@nospam.com> wrote:
> Hi What is a good way to respond to a failed Malloc (returns Null)? I do 
> not want to call Abort or Exit immediately.

Strategies for out of memory (OOM) handling are a fairly tough problem, with a fairly
broad scope.

Calling exit is not adequate in nontrivial programs, because the module in which OOM
occured doesn't have the authority to shut down the program, let alone abruptly.

One way is to implement exception handling, so that your code can take recovery actions
in an orderly way based on speculations along the lines of "what if out of memory happens
in this dynamic contour? or this one?"

You can set a side a chunk of reserve memory so that you can when OOM happens, it's actually
just a warning: the program is informed that OOM has occured, but is able to continue
thanks to the reserved memory (which is hopefully enough for recovery actions).
If the program is able to fully recover, so that it does not have to terminate, then
it can reset this machinery for the sake of future occurences.

No matter what you do, this is difficult to *test*, because programs make memory allocation
requests in many places and in many states.  So the different possible failure scenarios
are huge. What if the first N allocation requests succeed, and the N+1-th fails?
Now consider that for a large range of N, multiplied by variation in the
program's execution based on inputs ...

In general, all error checking and recovery type code which rarely executes only under
special conditions is difficult to test. If it is not tested (or else formally
verified), it only expresses the coder's best intentions to do something about
the situation. :)

-- 
If you ever need any coding done, I'm your goto man!

[toc] | [prev] | [next] | [standalone]

#26267

On 9/12/2012 4:07 PM, Eric wrote:
> Hi What is a good way to respond to a failed Malloc (returns Null)? I do
> not want to call Abort or Exit immediately.

     (Aside: You mean `malloc' and `NULL' and `abort' and `exit';
letter case matters in C.)

     Usually, a malloc() failure means the program will not be
able to continue: It wouldn't have requested the memory if it
weren't needed, and if it's needed but not there the program is
in trouble.  However, this does not mean the program must or
should halt abruptly: It may want to do some kind of "orderly
shutdown" before quitting, like saving in-memory data to files.
To make this work, the function whose malloc() call failed should
do its own cleanup, then report "*I* failed" to its caller, which
will do its bit of cleanup, then report failure to its caller, and
so on upward to a function that's at a high enough level to make
reasonable decisions on behalf of the entire program.  (Variations
of this scheme can involve setjmp() and longjmp(), but they have
their own perils and are not to be used lightly.)

     Sometimes the orderly shutdown activities will themselves need
dynamic memory, which may be in short supply if malloc() is failing.
One strategy to supply this need is to malloc() a chunk of "In Case
Of Emergency Break Glass" memory at program startup, then free()
it after malloc() failure to give the last-gasp code some room
to breathe.  I'd be tempted to package this inside a wrapper
to be called instead of malloc(), e.g.

	void *myMalloc(size_t bytes) {
	    static int initialized;
	    static void *emergency_stash;

	    if (!initialized) {
	        // Reserve a stash on the first call.
	        emergency_stash = malloc(STASH_SIZE);
	        if (emergency_stash == NULL) {
	            // Ooh, bad: Couldn't even get started!
	            return NULL;
	        }
	        initialized = 1;
	    }

	    void *new = malloc(bytes);
	    if (new == NULL && bytes > 0) {
	        // Trouble! Raid the rainy day fund -- not for
	        // this failing request, but for future requests
	        // the shutdown code may make.
	        free(emergency_stash);
	        emergency_stash = NULL;  // Don't double-free.
	    }
	    return new;
	}

     Some programs can proceed normally even after malloc() failure,
perhaps giving up on a sub-task but keeping the program alive
to do others.  "Sorry, Dave, I haven't got enough memory to load
yet another cute kitten video right now; maybe if you closed a
few that you've already got open we could try again."

-- 
Eric Sosman
esosman@ieee-dot-org.invalid
"The speed at which the system fails is usually not important."

[toc] | [prev] | [next] | [standalone]

#26271

Eric <nospam@nospam.com> writes:

> Hi What is a good way to respond to a failed Malloc (returns Null)? I do 
> not want to call Abort or Exit immediately.

Your options are to either end your program, decide you don't really
need the memory you just tried to allocate after all, or free() something
else up so you can use it.

But...  if you're on a 32 bit machine, you can malloc() a *lot* of
memory before it'll fail.  Are you sure you haven't corrupted the free
space pool?

[toc] | [prev] | [next] | [standalone]

#26278

On Sep 12, 11:07 pm, Joe Pfeiffer <pfeif...@cs.nmsu.edu> wrote:

> But...  if you're on a 32 bit machine, you can malloc() a *lot* of
> memory before it'll fail.  Are you sure you haven't corrupted the free
> space pool?

If you are on a 64 bit machine, you can't malloc() enough memory to
make it fail. You have some amount of RAM, and you have a lot lot more
swap space. malloc () would only fail if you run out of swap space.
But you never get that far, because using huge amounts of swap space
is so slow, the user dies of old age before malloc () fails.

(Situation may be different for users with tiny SSD drives)

[toc] | [prev] | [next] | [standalone]

#26279

On 9/12/2012 4:07 PM, Eric wrote:
> Hi What is a good way to respond to a failed Malloc (returns Null)? I do
> not want to call Abort or Exit immediately.

It all depends on what you needed the memory for.

For example, in one of the programs I support, memory is allocated for 
temporary sorting space.  As long as malloc() succeeded to get at least one 
buffer, a future failure is simply treated as "stop allocating buffers and 
use the ones you already have".  (There is also a user-configurable limit to 
the amount of memory it will even attempt to allocate.)

If the memory allocation was critical, then there's not much else you can do 
other than free up some other memory and hope it helps, or perhaps swap 
something out to disk (you still need to handle errors there, of course) and 
reuse that memory.

-- 
Kenneth Brody

[toc] | [prev] | [next] | [standalone]

#26281

In article <k2qq2i$aba$1@speranza.aioe.org>, Eric <nospam@nospam.com> wrote:

> Hi What is a good way to respond to a failed Malloc (returns Null)? I do 
> not want to call Abort or Exit immediately.

What do you want to do? Do that.

-- 
My name Indigo Montoya.  \\        Annoying Usenet one post at a time.
You flamed my father.     \'         At least I can stay in character.
Prepare to be spanked.   //               When you look into the void,
Stop posting that!      `/  the void looks into you, and fulfills you.

[toc] | [prev] | [next] | [standalone]

#26289

בתאריך יום רביעי, 12 בספטמבר 2012 21:07:51 UTC+1, מאת Eric:
> Hi What is a good way to respond to a failed Malloc (returns Null)? I do 
> not want to call Abort or Exit immediately.
> 
If there's a "natural" error return, pass it up to caller. So if you're returning a pointer to dynamic memory, return null.
If there's not, you have to decide whether it's worth complicating the interface. Normally the answer is yes, so it's traditional to return -1. Then caller decides what to do. 
So typically an allocation failure deep down in the program will be passed up
to main(). main() will then print out "Out of memory" and either abort or
skip to the next item in its list.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c

csiph-web