Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #67066 > unrolled thread

trouble of static working-data-buffers size assumption

Back to article view | Back to comp.lang.c

Contents

  trouble of static working-data-buffers size assumption fir <profesor.fir@gmail.com> - 2015-08-10 07:42 -0700
    Re: trouble of static working-data-buffers size assumption fir <profesor.fir@gmail.com> - 2015-08-10 07:48 -0700
      Re: trouble of static working-data-buffers size assumption Ken Brody <kenbrody@spamcop.net> - 2015-08-10 13:05 -0400
        Re: trouble of static working-data-buffers size assumption fir <profesor.fir@gmail.com> - 2015-08-10 10:21 -0700
          Re: trouble of static working-data-buffers size assumption fir <profesor.fir@gmail.com> - 2015-08-10 10:52 -0700
        Re: trouble of static working-data-buffers size assumption fir <profesor.fir@gmail.com> - 2015-08-10 11:25 -0700
          Re: trouble of static working-data-buffers size assumption Ken Brody <kenbrody@spamcop.net> - 2015-08-11 11:25 -0400
        Re: trouble of static working-data-buffers size assumption Richard Damon <Richard@Damon-Family.org> - 2015-08-10 22:36 -0400
          Re: trouble of static working-data-buffers size assumption Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-08-10 22:54 -0700
            Re: trouble of static working-data-buffers size assumption Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-08-11 13:41 +0100
              Re: trouble of static working-data-buffers size assumption Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-08-11 08:27 -0700
    Re: trouble of static working-data-buffers size assumption David Brown <david.brown@hesbynett.no> - 2015-08-10 18:22 +0200

#67066 — trouble of static working-data-buffers size assumption

I like static working buffers, sometimes 
you need some temporary working ram to make some
processing somewhat aside the main program data structures

I like them static becouse it is simpler, i do need manage its runtime creation and releasing, it is also a bit faster but the first point seem more important

I may have increasing amount of such eorking buffers to use when program grow bigger it may be counted in tens, though they are usually small
(in range of tens of hundreds kilobytes)

The trouble is yet that sometimes to state this fixed size i must do some assumption - i usuealli give wide marigin but it eventually may break;
sometimes for example i assume that client size of may window is less than 4k x 3k pixels,
(this is somewhat weird assumption )

yet worse i got this assumptions undocumented 
as this is usuall trade off - esability of working on it and unfaliability against more 
work - i would prefer and value mostly those solutions that gave both for free (simplicity and easability stays easy, unfaliables stays unfaliable)

some hints or thoughts?

(primitive insulters get off)

[toc] | [next] | [standalone]

#67067

W dniu poniedziałek, 10 sierpnia 2015 16:42:17 UTC+2 użytkownik fir napisał:

errata, should be : 
> as this is usuall trade off - esability of 
working on it against more 
> work and unfaliability 

yet ona advantage of static buffers (aside to mentioned simplicity and speed) is that realloc cannot fail (this is important as i seen many apps not ready for working with no more ram condition, and that was unpleasant,
app workng on static buffers would probably 
run further (if system would not break on its side) )

[toc] | [prev] | [next] | [standalone]

#67077

On 8/10/2015 10:48 AM, fir wrote:
> W dniu poniedziałek, 10 sierpnia 2015 16:42:17 UTC+2 użytkownik fir napisał:
>
> errata, should be :
>> as this is usuall trade off - esability of
> working on it against more
>> work and unfaliability
>
> yet ona advantage of static buffers (aside to mentioned simplicity and speed) is that realloc cannot fail (this is important as i seen many apps not ready for working with no more ram condition, and that was unpleasant,
> app workng on static buffers would probably
> run further (if system would not break on its side) )

Well, if realloc() fails because of insufficient memory, what makes you 
think that there's enough memory for the largest-possible static buffer?

I usually just use malloc/realloc, and keep track of my current allocation 
size.  I then have a quick to see if the current buffer is big enough, and 
realloc if not, storing the new size.  (Sometimes, I'll realloc to X bytes 
larger than the currently-required amount, to reduce realloc calls and 
possible memory fragmentation.)

No matter how big you make the static buffer, there will be some scenario 
where it's not large enough.  What do you do then?

-- 
Kenneth Brody

[toc] | [prev] | [next] | [standalone]

#67079

W dniu poniedziałek, 10 sierpnia 2015 19:06:10 UTC+2 użytkownik Ken Brody napisał:
> On 8/10/2015 10:48 AM, fir wrote:
> > W dniu poniedziałek, 10 sierpnia 2015 16:42:17 UTC+2 użytkownik fir napisał:
> >
> > errata, should be :
> >> as this is usuall trade off - esability of
> > working on it against more
> >> work and unfaliability
> >
> > yet ona advantage of static buffers (aside to mentioned simplicity and speed) is that realloc cannot fail (this is important as i seen many apps not ready for working with no more ram condition, and that was unpleasant,
> > app workng on static buffers would probably
> > run further (if system would not break on its side) )
> 
> Well, if realloc() fails because of insufficient memory, what makes you 
> think that there's enough memory for the largest-possible static buffer?
> 

maybe i do not quite understand this question 
(I also may be a bit wrong in understanding 
memory behaviours, as i code only c apps for windows  im saying only windows here)
but how i see it:
If you make whole app with static buffers
(on reasonable overall size i dont know like 700 MB) and it will run, even if other apps will drain out the memory this app shopuld 
probably still work (thought it may maybe heve trouble to recover from swap if swapped)
If you use mallocs and reallocs the heavy amount of them you got in code will return errors to you (its unlike for anybody (and probably nonsense) to code recovery behaviuors
for all this cases to this app will be broken
(probably the most reasonable behaviour of that would be show message box "out of memory, please free some system memory", with a button "retry" - then continue - well if this work that would have some sense but most apps dont do that.. depending how app with only static buffers will behave in system where ram is drained usage of such static buffers may be advantage or no)

As to how resolve a problem with static buffers 'size assumption' this is the question i wrote






> I usually just use malloc/realloc, and keep track of my current allocation 
> size.  I then have a quick to see if the current buffer is big enough, and 
> realloc if not, storing the new size.  (Sometimes, I'll realloc to X bytes 
> larger than the currently-required amount, to reduce realloc calls and 
> possible memory fragmentation.)
> 
> No matter how big you make the static buffer, there will be some scenario 
> where it's not large enough.  What do you do then?
> 

[toc] | [prev] | [next] | [standalone]

#67081

W dniu poniedziałek, 10 sierpnia 2015 19:21:50 UTC+2 użytkownik fir napisał:
> As to how resolve a problem with static buffers 'size assumption' this is the question i wrote
> 
> 

I at least put runtime checks but this is still sad when someone will run app in larger input condition and it will show an assert with error instead working... The cure seem to be just use reallocks in all that cases but is more work (and for ma look a bit uglier).. im not sure.. maybe i will

[toc] | [prev] | [next] | [standalone]

#67083

W dniu poniedziałek, 10 sierpnia 2015 19:06:10 UTC+2 użytkownik Ken Brody napisał:
> On 8/10/2015 10:48 AM, fir wrote:
> > W dniu poniedziałek, 10 sierpnia 2015 16:42:17 UTC+2 użytkownik fir napisał:
> >
> > errata, should be :
> >> as this is usuall trade off - esability of
> > working on it against more
> >> work and unfaliability
> >
> > yet ona advantage of static buffers (aside to mentioned simplicity and speed) is that realloc cannot fail (this is important as i seen many apps not ready for working with no more ram condition, and that was unpleasant,
> > app workng on static buffers would probably
> > run further (if system would not break on its side) )
> 
> Well, if realloc() fails because of insufficient memory, what makes you 
> think that there's enough memory for the largest-possible static buffer?
> 
> I usually just use malloc/realloc, and keep track of my current allocation 
> size.  I then have a quick to see if the current buffer is big enough, and 
> realloc if not, storing the new size.  (Sometimes, I'll realloc to X bytes 
> larger than the currently-required amount, to reduce realloc calls and 
> possible memory fragmentation.)
> 

what memopry fragmentation? realloc will give you solid ram chunk, afaik no trouble with it..


> No matter how big you make the static buffer, there will be some scenario 
> where it's not large enough.  What do you do then?
> 
well this leeds me to reallocs..
but in reallity it is some trade-off assumptions for reallocs, not 100% clear choice.. though im like to seem change to reallocs (last few years i prefered statics and assumptions now wonder if to change side)

[toc] | [prev] | [next] | [standalone]

#67113

On 8/10/2015 2:25 PM, fir wrote:
> W dniu poniedziałek, 10 sierpnia 2015 19:06:10 UTC+2 użytkownik Ken Brody
> napisał:
[...]
>> I usually just use malloc/realloc, and keep track of my current
>> allocation size.  I then have a quick to see if the current buffer is
>> big enough, and realloc if not, storing the new size.  (Sometimes, I'll
>> realloc to X bytes larger than the currently-required amount, to reduce
>> realloc calls and possible memory fragmentation.)
>>
>
> what memopry fragmentation? realloc will give you solid ram chunk, afaik
> no trouble with it..

What happens when you try to realloc the buffer larger, but malloc has 
already allocated the memory immediately following the current buffer?  It 
must get you a new buffer.  Then, when you realloc again for a larger 
buffer, it's possible that the memory immediately following the new buffer 
is also already allocated, so another buffer needs to be returned.  (This is 
probably more noticeable when starting with small buffers, with small 
increments in size.)

You now have chunks of allocated memory with small chunks of now-free 
memory, which may or may not ever get allocated, scattered amongst them.

I didn't mean that realloc() would return a "fragmented" buffer (such as 
disk storage), but rather that the malloc "heap" would become fragmented.

>> No matter how big you make the static buffer, there will be some
>> scenario where it's not large enough.  What do you do then?
>>
> well this leeds me to reallocs.. but in reallity it is some trade-off
> assumptions for reallocs, not 100% clear choice.. though im like to seem
> change to reallocs (last few years i prefered statics and assumptions now
> wonder if to change side)

Well, you still have to handle the case of "I need more memory than the 
static buffer I reserved".

-- 
Kenneth Brody

[toc] | [prev] | [next] | [standalone]

#67101

On 8/10/15 1:05 PM, Ken Brody wrote:
> On 8/10/2015 10:48 AM, fir wrote:
>> W dniu poniedziałek, 10 sierpnia 2015 16:42:17 UTC+2 użytkownik fir
>> napisał:
>>
>> errata, should be :
>>> as this is usuall trade off - esability of
>> working on it against more
>>> work and unfaliability
>>
>> yet ona advantage of static buffers (aside to mentioned simplicity and
>> speed) is that realloc cannot fail (this is important as i seen many
>> apps not ready for working with no more ram condition, and that was
>> unpleasant,
>> app workng on static buffers would probably
>> run further (if system would not break on its side) )
>
> Well, if realloc() fails because of insufficient memory, what makes you
> think that there's enough memory for the largest-possible static buffer?
>
> I usually just use malloc/realloc, and keep track of my current
> allocation size.  I then have a quick to see if the current buffer is
> big enough, and realloc if not, storing the new size.  (Sometimes, I'll
> realloc to X bytes larger than the currently-required amount, to reduce
> realloc calls and possible memory fragmentation.)
>
> No matter how big you make the static buffer, there will be some
> scenario where it's not large enough.  What do you do then?
>

The concept of making the data statically allocated is that you evaluate 
your requirements and allocate enough memory to meet your requirements. 
If you don't have enough memory, you find out at link time (or at the 
very latest, loading into memory time). If a request comes in that you 
don't have enough memory for, it means your requirements have been 
exceeded, and you fall back to your defined way to reject/delay the 
request. (If your requirements haven't been exceeded then you made a 
mistake in your allocations).

What this avoids happening is a request which exceeds your requirements, 
and thus being rejectable, from taking resources that are needed for a 
request that is within your requirements that you aren't doing at the 
moment.

Static allocation primarily is used when you CAN precompute the amount 
of required buffers, though you can combine the static buffers with 
dynamic memory buffers to allow non/beyond required requests to be 
attempted.

[toc] | [prev] | [next] | [standalone]

#67105

On Tuesday, August 11, 2015 at 3:36:47 AM UTC+1, Richard Damon wrote:
>
> The concept of making the data statically allocated is that you evaluate 
> your requirements and allocate enough memory to meet your requirements. 
> If you don't have enough memory, you find out at link time (or at the 
> very latest, loading into memory time). If a request comes in that you 
> don't have enough memory for, it means your requirements have been 
> exceeded, and you fall back to your defined way to reject/delay the 
> request. (If your requirements haven't been exceeded then you made a 
> mistake in your allocations).
> 
> What this avoids happening is a request which exceeds your requirements, 
> and thus being rejectable, from taking resources that are needed for a 
> request that is within your requirements that you aren't doing at the 
> moment.
> 
> Static allocation primarily is used when you CAN precompute the amount 
> of required buffers, though you can combine the static buffers with 
> dynamic memory buffers to allow non/beyond required requests to be 
> attempted.
<
Though the paradigm is

TOUR *tspsetup(int N)
{
   TOUR *answer = malloc(sizeof(TOUR));
   if(!answer)
       goto out_ofmemory;
   answer->whizzymatrix = 0;
   answer->whizzytourbuffer = 0;

   answer->whizzymatrix = malloc(N * N * sizeof(CELL));
   if(!answer->whizzymatrix)
     goto out_of_memory;
   answer->whizzytourbuffer = malloc(N *  N * N * sizeof(WHIZZYTOUR));
   if(!answer->whizzytour_buffer)
       goto out_of_memory;

  out_of_memory;
     killtour(answer);
     return 0;
}

So we allocate our workspace when N comes in, and if any of the
allocations fail, we free everything up and return failure.

Where it won't work is where our TSP algorithm requires that we 
constantly grow and prune some sort of tour graph, with no low
limit on the maximum size (maybe because the graph is reasonable
size for typical city problems, but becomes N! for a few very
difficult ones with lots of approximately equal sub-tours, it's
an NP-hard problem, after all). But a  static buffer isn't going 
to address that problem either.

[toc] | [prev] | [next] | [standalone]

#67109

Malcolm McLean <malcolm.mclean5@btinternet.com> writes:
<snip>
> Though the paradigm is
>
> TOUR *tspsetup(int N)
> {
>    TOUR *answer = malloc(sizeof(TOUR));
>    if(!answer)
>        goto out_ofmemory;
>    answer->whizzymatrix = 0;
>    answer->whizzytourbuffer = 0;
>
>    answer->whizzymatrix = malloc(N * N * sizeof(CELL));
>    if(!answer->whizzymatrix)
>      goto out_of_memory;
>    answer->whizzytourbuffer = malloc(N *  N * N * sizeof(WHIZZYTOUR));
>    if(!answer->whizzytour_buffer)
>        goto out_of_memory;

     return answer;
    
>
>   out_of_memory;
(: rather than ;)
>      killtour(answer);
>      return 0;
> }

Alternatively (just so that it's clear there are other ways):

  TOUR *tpsetup(size_t N)
  {
      TOUR *answer = malloc(sizeof *answer);
      if (answer) {
          answer->whizzymatrix =
              malloc(N * N * sizeof *answer->whizzymatrix);
          answer->whizzytourbuffer =
              malloc(N * N * N * sizeof *answer->whizztourbuffer);
          if (answer->whizzymatrix && answer->whizzytourbuffer)
              return answer;
      }
      killtour(answer);
      return 0;
  }

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#67114

On Tuesday, August 11, 2015 at 1:41:55 PM UTC+1, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.mclean5@btinternet.com> writes:
>
> Alternatively (just so that it's clear there are other ways):
> 
>   TOUR *tpsetup(size_t N)
>   {
>       TOUR *answer = malloc(sizeof *answer);
>       if (answer) {
>           answer->whizzymatrix =
>               malloc(N * N * sizeof *answer->whizzymatrix);
>           answer->whizzytourbuffer =
>               malloc(N * N * N * sizeof *answer->whizztourbuffer);
>           if (answer->whizzymatrix && answer->whizzytourbuffer)
>               return answer;
>       }
>       killtour(answer);
>       return 0;
>   }
> 
But let's say we swap the allocations to whizzymatrix and whizzytourbuffer.
The whizzytourbuffer is N*N*N. Suppose that fails. N*N is obviously still quite
a large allocation. We're tying up those resources, albeit only for a fraction of
a second until the killtour function frees them.

It's better to go into error-handling code immediately on allocation failure.

[toc] | [prev] | [next] | [standalone]

#67074

On 10/08/15 16:42, fir wrote:
> I like static working buffers, sometimes 
> you need some temporary working ram to make some
> processing somewhat aside the main program data structures
> 
> I like them static becouse it is simpler, i do need manage its runtime creation and releasing, it is also a bit faster but the first point seem more important
> 
> I may have increasing amount of such eorking buffers to use when program grow bigger it may be counted in tens, though they are usually small
> (in range of tens of hundreds kilobytes)
> 
> The trouble is yet that sometimes to state this fixed size i must do some assumption - i usuealli give wide marigin but it eventually may break;
> sometimes for example i assume that client size of may window is less than 4k x 3k pixels,
> (this is somewhat weird assumption )
> 
> yet worse i got this assumptions undocumented 
> as this is usuall trade off - esability of working on it and unfaliability against more 
> work - i would prefer and value mostly those solutions that gave both for free (simplicity and easability stays easy, unfaliables stays unfaliable)
> 
> some hints or thoughts?
> 
> (primitive insulters get off)
> 

The usual way is to do something like this:

#define maxWindowX 4096
#define maxWindowY 3072

typedef struct { ..... } pixel;
static pixel window[maxX][maxY];

Put your size define's in an easily identifiable configuration file, and
make all your static data's size depend on those configuration values.
Then you have assumptions about maximum sizes, but at least they are
clearly documented.

(In my line of work, static buffers are the norm - dynamic memory is
avoided whenever possible.  But usually I have the advantage of knowing
exactly what hardware the code will run on.)


Another possibility is to use dynamic memory, but allocate it once on
startup.  You don't have to worry about the success or failure of
allocations (after the initialisation routines), or ownership of memory,
or freeing it (the OS handles that at program exit).

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web