Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #11948 > unrolled thread

Competing criteria

Back to article view | Back to comp.lang.c

Contents

  Competing criteria nroberts <roberts.noah@gmail.com> - 2011-10-21 14:49 -0700
    Re: Competing criteria Seebs <usenet-nospam@seebs.net> - 2011-10-21 22:16 +0000
      Re: Competing criteria Roberto Waltman <usenet@rwaltman.com> - 2011-10-22 14:06 -0400
        Re: Competing criteria Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-22 16:47 -0400
          Re: Competing criteria Ark <akhasin@macroexpressions.com> - 2011-10-22 17:33 -0400
            Re: Competing criteria Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-22 18:20 -0400
              Re: Competing criteria Ark <akhasin@macroexpressions.com> - 2011-10-22 21:29 -0400
              Re: Competing criteria "io_x" <a@b.c.invalid> - 2011-10-23 10:32 +0200
                Re: Competing criteria "Joachim Schmitz" <nospam.jojo@schmitz-digital.de> - 2011-10-23 11:06 +0200
                  Re: Competing criteria Willem <willem@toad.stack.nl> - 2011-10-23 09:09 +0000
                    Re: Competing criteria "io_x" <a@b.c.invalid> - 2011-10-23 12:04 +0200
                      Re: Competing criteria "io_x" <a@b.c.invalid> - 2011-10-27 08:44 +0200
                  Re: Competing criteria Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-10-23 05:01 -0700
              Re: Competing criteria Eric Sosman <esosman@ieee-dot-org.invalid> - 2011-10-23 08:29 -0400
                Re: Competing criteria Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-23 09:15 -0400
                  Re: Competing criteria Malcolm McLean <malcolm.mclean5@btinternet.com> - 2011-10-23 06:24 -0700
                    Re: Competing criteria Markus Wichmann <nullplan@gmx.net> - 2011-10-24 22:39 +0200
                      Re: Competing criteria Keith Thompson <kst-u@mib.org> - 2011-10-24 17:10 -0700
                      Re: Competing criteria Phil Carmody <thefatphil_demunged@yahoo.co.uk> - 2011-10-25 13:09 +0300
                      Re: Competing criteria Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-10-26 00:23 -0700
        Re: Competing criteria Phil Carmody <thefatphil_demunged@yahoo.co.uk> - 2011-10-23 00:02 +0300
        Re: Competing criteria Seebs <usenet-nospam@seebs.net> - 2011-10-24 20:10 +0000
          Re: Competing criteria Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-24 19:36 -0400
          Re: Competing criteria nroberts <roberts.noah@gmail.com> - 2011-10-25 10:07 -0700
            Re: Competing criteria Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-25 16:51 -0400
    Re: Competing criteria John Gordon <gordon@panix.com> - 2011-10-21 22:37 +0000
    Re: Competing criteria Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-21 20:42 -0400

Page 2 of 2 — ← Prev page 1 [2]

#11968

Roberto Waltman <usenet@rwaltman.com> writes:
> Seebs  wrote:
> >  the canonical
> >solution to "not heap allocation" is to define a reasonable maximum count
> >of simultaneous objects, statically allocate them, and then basically
> >write your own mini-allocator that keeps track of which ones you're using.
> >This, you will note, is probably stupid.
> 
> Far from being stupid, that is the only way you can guarantee that
> memory allocation will not fail due to memory fragmentation, in
> systems with limited resources, no virtual memory and that must run
> 24/7.

False, and likely to fail in most embedded scenarios I've worked on.
Likely to fail, as if you have 100 units of RAM, and may need up to 20
of size 5, or 5 of size 20, but not simultaniously, then you must use
dynamic allocation.

And false, because if you never hand out pointers, but handles, then
you can use dynamic memory allocation and compact your heap either in
the background or on demand.

Phil
-- 
Unix is simple. It just takes a genius to understand its simplicity
-- Dennis Ritchie (1941-2011), Unix Co-Creator

[toc] | [prev] | [next] | [standalone]

#12053

On 2011-10-22, Roberto Waltman <usenet@rwaltman.com> wrote:
> Seebs  wrote:
>>  the canonical
>>solution to "not heap allocation" is to define a reasonable maximum count
>>of simultaneous objects, statically allocate them, and then basically
>>write your own mini-allocator that keeps track of which ones you're using.
>>This, you will note, is probably stupid.

> Far from being stupid, that is the only way you can guarantee that
> memory allocation will not fail due to memory fragmentation, in
> systems with limited resources, no virtual memory and that must run
> 24/7.

True, but it doesn't prevent memory allocation from failing due to a lack
of resources.  So you still have to deal with memory allocation failures.

And given a choice between rare failures due to fragmentation, and frequent
failures due to poor choice of static limits...

-s
-- 
Copyright 2011, all wrongs reversed.  Peter Seebach / usenet-nospam@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.

[toc] | [prev] | [next] | [standalone]

#12061

On 10/24/11 4:10 PM, Seebs wrote:
> On 2011-10-22, Roberto Waltman<usenet@rwaltman.com>  wrote:
>> Seebs  wrote:
>>>   the canonical
>>> solution to "not heap allocation" is to define a reasonable maximum count
>>> of simultaneous objects, statically allocate them, and then basically
>>> write your own mini-allocator that keeps track of which ones you're using.
>>> This, you will note, is probably stupid.
>
>> Far from being stupid, that is the only way you can guarantee that
>> memory allocation will not fail due to memory fragmentation, in
>> systems with limited resources, no virtual memory and that must run
>> 24/7.
>
> True, but it doesn't prevent memory allocation from failing due to a lack
> of resources.  So you still have to deal with memory allocation failures.
>
> And given a choice between rare failures due to fragmentation, and frequent
> failures due to poor choice of static limits...
>
> -s

If you have frequent failures, you didn't set your static limits 
correctly. To do the static preallocations, you need to do analysis. You 
need to define what is allowed to be active at any given time, an make 
sure you have sufficient resources statically allocated to meet that 
demand. if you can't do the analysis, you can't make the performance 
guarantee. If you can't preallocate sufficient buffers to meet that 
demand, you need to do one of the following:

1) Find a way to do some of the jobs with less memory.
2) Add more memory to the system.
3) Reduce the required guaranteed performance

If you switch to a dynamic allocation system, it is virtually impossible 
to really guarantee a performance level, and it can look really bad for 
you to have a device whose performance deteriorates over time and 
requires periodic reboots to keep it working up to spec.

To have a allocation failure, because you are operating beyond your 
spec, is NOT a system failure, it should be allowable for you to refuse 
to process that request.

If you have some memory left over, you might want to switch to making a 
heap request when your preallocated buffers are exhausted, in this case 
since you are now allowed to fail, you are ok. When done, you should 
know that THIS buffer should be returned to the heap and not one of the 
other preallocated buffer to avoid heap fragmentation (and depending on 
how you did it, the preallocated buffers may not have come from the heap).

Note, that it might be possible that you know that the system goes into 
different "modes" where the types of buffers needed are different. This 
can make preallocation a bit harder, but judicious use of unions can 
often solve this. To the degree that you can't preallocate as 
efficiently as you think you might be able to use the heap, consider 
that the cost for getting the guarantee. To really be sure the heap will 
work would add a LOT of stress testing and analysis, and even then there 
will be some doubt. One way to think about it, which car would you buy, 
the one guaranteed to be able to get up to 70 mph, and if conditions are 
good a bit beyond that, or the car that if conditions are right will 
surpass 100 mph, but they can't be sure if there might be cases, even 
somewhat rare (like maybe driving for a few hours), where it will only 
hit 45 mph. If I plan to be driving on the highway, I don't think I 
would want that second car. It may even "on average" go faster, but when 
I want the performance, I want it. In the same way, if I buy a system 
with a spec, I want it to meet the spec, not occasionally come up short 
needing a reboot to clear up its memory fragmentation.

[toc] | [prev] | [next] | [standalone]

#12089

On Oct 24, 1:10 pm, Seebs <usenet-nos...@seebs.net> wrote:
> On 2011-10-22, Roberto Waltman <use...@rwaltman.com> wrote:
>
> > Seebs  wrote:
> >>  the canonical
> >>solution to "not heap allocation" is to define a reasonable maximum count
> >>of simultaneous objects, statically allocate them, and then basically
> >>write your own mini-allocator that keeps track of which ones you're using.
> >>This, you will note, is probably stupid.
> > Far from being stupid, that is the only way you can guarantee that
> > memory allocation will not fail due to memory fragmentation, in
> > systems with limited resources, no virtual memory and that must run
> > 24/7.
>
> True, but it doesn't prevent memory allocation from failing due to a lack
> of resources.  So you still have to deal with memory allocation failures.
>
> And given a choice between rare failures due to fragmentation, and frequent
> failures due to poor choice of static limits...

This is sort of where I am.

Most high-availability systems developers don't limit themselves to
not using heap memory, right?  It seems to me that in systems like
that, which are on large, fast servers with lots of memory, shouldn't
run into memory issues a lot.  Or am I mistaken?

I mean, the front end for this service is a soap server.  Certainly
the likes of Tomcat are using heap memory... right?

It seems to me that I should be more worried about robustness than
what appears to me an unlikely scenario in which memory fragmentation
keeps me from allocating a 100 byte string.  I'd also think that
maintenance and simplicity should be a higher concern.

I understand what some are saying here, how even in hosted
environments malloc calls can fail...but how often does this really
come up?  I've never seen it happen when there wasn't a deeper,
underlying issue causing a loop to run amok or some kind of
ridiculously erroneous size request.  In my experience anyway, you're
more likely to run into a bug in your code causing a cascade failure
than malloc returning null.

[toc] | [prev] | [next] | [standalone]

#12099

On 10/25/11 1:07 PM, nroberts wrote:
> On Oct 24, 1:10 pm, Seebs<usenet-nos...@seebs.net>  wrote:
>> On 2011-10-22, Roberto Waltman<use...@rwaltman.com>  wrote:
>>
>>> Seebs  wrote:
>>>>   the canonical
>>>> solution to "not heap allocation" is to define a reasonable maximum count
>>>> of simultaneous objects, statically allocate them, and then basically
>>>> write your own mini-allocator that keeps track of which ones you're using.
>>>> This, you will note, is probably stupid.
>>> Far from being stupid, that is the only way you can guarantee that
>>> memory allocation will not fail due to memory fragmentation, in
>>> systems with limited resources, no virtual memory and that must run
>>> 24/7.
>>
>> True, but it doesn't prevent memory allocation from failing due to a lack
>> of resources.  So you still have to deal with memory allocation failures.
>>
>> And given a choice between rare failures due to fragmentation, and frequent
>> failures due to poor choice of static limits...
>
> This is sort of where I am.
>
> Most high-availability systems developers don't limit themselves to
> not using heap memory, right?  It seems to me that in systems like
> that, which are on large, fast servers with lots of memory, shouldn't
> run into memory issues a lot.  Or am I mistaken?
>
> I mean, the front end for this service is a soap server.  Certainly
> the likes of Tomcat are using heap memory... right?
>
> It seems to me that I should be more worried about robustness than
> what appears to me an unlikely scenario in which memory fragmentation
> keeps me from allocating a 100 byte string.  I'd also think that
> maintenance and simplicity should be a higher concern.
>
> I understand what some are saying here, how even in hosted
> environments malloc calls can fail...but how often does this really
> come up?  I've never seen it happen when there wasn't a deeper,
> underlying issue causing a loop to run amok or some kind of
> ridiculously erroneous size request.  In my experience anyway, you're
> more likely to run into a bug in your code causing a cascade failure
> than malloc returning null.

Note that this thread started on a design rule for *embedded* systems of 
not using the heap. Once you get out of the embedded world, and for this 
purposes I suppose the line is somewhere near having mass-storage and 
virtual memory, the game changes. Here, normally long before you 
actually run out of heap, other issues tend to affect your system 
performance.

[toc] | [prev] | [next] | [standalone]

#11950

In <b512b55a-437a-4ba2-97cd-c483e7449492@g7g2000vbv.googlegroups.com> nroberts <roberts.noah@gmail.com> writes:

> At the same time he wants me to write reusable code and has previously
> wanted me to make unit tests, though now I'm not even sure about that
> since he won't even look at them.

He wants to ensure that adequate tests exist but doesn't care about the
gory details; that's not necessarily an unreasonable attitude for a project
lead.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon@panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

[toc] | [prev] | [next] | [standalone]

#11952

On 10/21/11 5:49 PM, nroberts wrote:
> I'm having trouble figuring out how to get seemingly mutually
> exclusive goals and criteria to work together.
>
> The technical lead/architect for the project I'm on is from an
> embedded background and therefore doesn't like heap allocation.  I
> don't know if this is a real issue in that domain, having never done
> it myself, but that's what he's told me when complaining about my use
> of allocated buffers and such.  Factory like functions also seem to
> upset him.  I'm not so sure about function pointers, but every time
> I've made something that wasn't of the direct call, stack or global
> variable kind of design he's kind of freaked out on me.
>
> At the same time he wants me to write reusable code and has previously
> wanted me to make unit tests, though now I'm not even sure about that
> since he won't even look at them.
>
> I'm getting kind of pissed off to tell the truth, but I still want to
> try.
>
> I'm coming into a project where the best solution seems to be a
> multithreaded service.  It's basically a proxy for a soap server
> created by a third party that we're making generic and less soapy in
> order for our various products to use it.  Some of the calls can take
> a long time and writing it multithreaded allows us to use the gsoap
> library without having to muck about to use it asynchronously as it
> doesn't really support such use.
>
> I created a prototype for this service in C++ using all the
> allocation, RAII, polymorphism, etc...so I could get a clear picture
> of the problem.  Now I need to convert the design into C (I'm fairly
> certain he's going to tell me to).  Normally this would be fairly
> easy, replace exception processing with error codes, rewrite std::list
> and such for all the types I want to hold or to use void*...use opaque
> types and createXXX() functions...  I'm not doing anything
> particularly incredible with the C++, it could almost as easily be C.
> This kind of stuff doesn't fly here though.
>
> I can create circular buffer stacks to handle things like my thread
> pools instead of allocating stack nodes.  I can just declare my opaque
> types openly and have initXXX() functions. Etc...I think this kind of
> thing would make the guy happy but I don't see how to do that while
> also meeting another goal he's said I need to meet: make this thing so
> that it's a generic architecture for any soap connecting proxy.  I
> can't think of how to do such circular buffer stacks for specific
> types (because I can't allocate) and still write them in a way that
> can be used in different ways.
>
> How does one make reusable components without at least some generics
> and heap allocations?  When everything has to be a direct call, how do
> you allow clients to override behavior?  What are some idioms,
> patterns, whatever that I can use to at least make some attempt at
> getting both of these ideals to work together?

As someone who has written a number of embedded projects, I can give 
some grounds for the desire to limit heap usage and where it can 
reasonably be used and where it should not.

I typical embedded project can NOT make the common "hosted" assumption 
of nearly unlimited memory being available. It also typically needs to 
have a virtually infinite run time, so heap fragmentation needs to be a 
concern.

The guidelines I tend to work with are:

1) Heap allocations at "start-up" are normally not an issue. If you 
don't have the memory needed, you find out right away, not at some 
unknown time in the future.

2) Critical tasks, that must be able to complete on a schedule may NOT 
make further heap allocation, but only use statically allocated buffers 
or buffer allocated at start-up.

2a) If a task only becomes "critical" after some other action that is 
allowed to fail, that other action is allowed to allocate the resources 
for the critical task, and must fail if it can't get them.

3) Non-critical tasks may be allowed to allocate dynamic resources, but 
there should be points in the execution of the program where most of 
these have been returned to the heap to clear up heap fragmentation.

With this, your program can be generic (at compile time), as hopefully 
you can know which types are needed for this program, and you can then 
preallocate the needed buffer set, but the code is still usable for 
other applications that use different sets of types.

If you point out that you are only doing heap allocations at "startup" 
time, (or for non-critical code tasks, that are allowed to fail/be 
delayed), then he will hopefully be satisfied.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c

csiph-web