Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #17472 > unrolled thread

how to figure out the size of buffer from malloc

Back to article view | Back to comp.lang.c

Contents

  how to figure out the size of buffer from malloc pembed2012 <once@was.enough.invalid> - 2012-02-16 20:21 +0000
    Re: how to figure out the size of buffer from malloc James Kuyper <jameskuyper@verizon.net> - 2012-02-16 15:38 -0500
    Re: how to figure out the size of buffer from malloc John Gordon <gordon@panix.com> - 2012-02-16 20:48 +0000
      Re: how to figure out the size of buffer from malloc pembed2012 <once@was.enough.invalid> - 2012-02-16 22:42 +0000
        Re: how to figure out the size of buffer from malloc John Gordon <gordon@panix.com> - 2012-02-16 22:50 +0000
        Re: how to figure out the size of buffer from malloc Keith Thompson <kst-u@mib.org> - 2012-02-16 14:50 -0800
          Re: how to figure out the size of buffer from malloc pembed2012 <once@was.enough.invalid> - 2012-02-17 19:45 +0000
            Re: how to figure out the size of buffer from malloc China Blue Sea <chine.bleu@yahoo.com> - 2012-02-17 11:53 -0800
            Re: how to figure out the size of buffer from malloc Ike Naar <ike@sverige.freeshell.org> - 2012-02-17 20:23 +0000
            Re: how to figure out the size of buffer from malloc James Kuyper <jameskuyper@verizon.net> - 2012-02-17 15:33 -0500
              Re: how to figure out the size of buffer from malloc pembed2012 <once@was.enough.invalid> - 2012-02-17 22:13 +0000
                Re: how to figure out the size of buffer from malloc James Kuyper <jameskuyper@verizon.net> - 2012-02-17 17:25 -0500
                Re: how to figure out the size of buffer from malloc Shao Miller <sha0.miller@gmail.com> - 2012-02-17 17:54 -0500
                Re: how to figure out the size of buffer from malloc Keith Thompson <kst-u@mib.org> - 2012-02-17 15:01 -0800
                  Re: how to figure out the size of buffer from malloc Shao Miller <sha0.miller@gmail.com> - 2012-02-17 18:10 -0500
                Re: how to figure out the size of buffer from malloc Ian Pilcher <arequipeno@gmail.com> - 2012-02-17 18:14 -0600
                Re: how to figure out the size of buffer from malloc "BartC" <bc@freeuk.com> - 2012-02-18 11:07 +0000
                Re: how to figure out the size of buffer from malloc ImpalerCore <jadill33@gmail.com> - 2012-02-18 04:40 -0800
                Re: how to figure out the size of buffer from malloc Kaz Kylheku <kaz@kylheku.com> - 2012-02-18 12:48 +0000
            Re: how to figure out the size of buffer from malloc Shao Miller <sha0.miller@gmail.com> - 2012-02-17 17:45 -0500
        Re: how to figure out the size of buffer from malloc Shao Miller <sha0.miller@gmail.com> - 2012-02-16 22:18 -0500
    Re: how to figure out the size of buffer from malloc Devil with the China Blue Dress <chine.bleu@yahoo.com> - 2012-02-16 13:20 -0800
    Re: how to figure out the size of buffer from malloc Lew Pitcher <lpitcher@teksavvy.com> - 2012-02-16 17:07 -0500
    Re: how to figure out the size of buffer from malloc Keith Thompson <kst-u@mib.org> - 2012-02-16 14:28 -0800
    Re: how to figure out the size of buffer from malloc Mark Adler <madler@alumni.caltech.edu> - 2012-02-16 23:49 -0800
    Re: how to figure out the size of buffer from malloc Michael Angelo Ravera <maravera@prodigy.net> - 2012-02-18 09:54 -0800
      Re: how to figure out the size of buffer from malloc gazelle@shell.xmission.com (Kenny McCormack) - 2012-02-18 18:21 +0000
        Re: how to figure out the size of buffer from malloc Shao Miller <sha0.miller@gmail.com> - 2012-02-18 13:56 -0500
        Re: how to figure out the size of buffer from malloc Stephen Sprunk <stephen@sprunk.org> - 2012-02-18 13:42 -0600

Page 1 of 2  [1] 2  Next page →

#17472 — how to figure out the size of buffer from malloc

I am wondering how does 'free' know how many bytes to free? For example:

char* s = (char*)malloc(10);
free(s); // how many bytes to free?

How does free know how many bytes to free? I have heard that malloc
also allocate additional memory to hold some header information. Where
is this header information stored? If we have access to the header, we
can find that out?

Thanks!

[toc] | [next] | [standalone]

#17473

On 02/16/2012 03:21 PM, pembed2012 wrote:
> I am wondering how does 'free' know how many bytes to free? For example:
> 
> char* s = (char*)malloc(10);
> free(s); // how many bytes to free?
> 
> How does free know how many bytes to free? I have heard that malloc
> also allocate additional memory to hold some header information. Where
> is this header information stored? If we have access to the header, we
> can find that out?

Howe that information is stored is an implementation detail, likely to
be different on different implementations. As a result, there's no
portable way you can obtain that information.

One common technique is to have malloc() allocate a little bit more than
the requested amount of memory, and to store the size of the allocation
in that extra piece of memory. This extra memory is usually stored at a
fixed negative offset from the pointer that you get from malloc().

Another technique is round each allocation request up to a power of 2,
and to maintain a separate pool of memory for each power of 2. By
looking at the address, free() can tell which pool it came from, and
therefore, how much memory to free.

There's several other, more sophisticated techniques that are in actual
use, but I hope that those two are sufficient for you to understand why
there's no portable way to get this information.
 --
James Kuyper

[toc] | [prev] | [next] | [standalone]

#17474

In <jhjofu$jer$1@speranza.aioe.org> pembed2012 <once@was.enough.invalid> writes:

> I am wondering how does 'free' know how many bytes to free? For example:

> char* s = (char*)malloc(10);
> free(s); // how many bytes to free?

A common method is for malloc to allocate a few extra bytes, store the
size in the extra space at the beginning, and return a pointer to the
remaining area.

In your sample code above, malloc might actually allocate fourteen bytes,
store the size information in the first four bytes and return a pointer
to the remaining ten bytes.

Then when free is called, it knows it can back up four bytes from the
given address to find the size of the area to be freed.

> How does free know how many bytes to free? I have heard that malloc
> also allocate additional memory to hold some header information. Where
> is this header information stored? If we have access to the header, we
> can find that out?

It's all compiler- and library-specific.  The language itself doesn't
specify how it is done.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon@panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

[toc] | [prev] | [next] | [standalone]

#17482

On Thu, 16 Feb 2012 20:48:06 +0000, John Gordon wrote:

> In <jhjofu$jer$1@speranza.aioe.org> pembed2012 <once@was.enough.invalid>
> writes:
> 
>> I am wondering how does 'free' know how many bytes to free? For
>> example:
> 
>> char* s = (char*)malloc(10);
>> free(s); // how many bytes to free?
> 
> A common method is for malloc to allocate a few extra bytes, store the
> size in the extra space at the beginning, and return a pointer to the
> remaining area.
> 
> In your sample code above, malloc might actually allocate fourteen
> bytes, store the size information in the first four bytes and return a
> pointer to the remaining ten bytes.
> 
> Then when free is called, it knows it can back up four bytes from the
> given address to find the size of the area to be freed.
> 
>> How does free know how many bytes to free? I have heard that malloc
>> also allocate additional memory to hold some header information. Where
>> is this header information stored? If we have access to the header, we
>> can find that out?
> 
> It's all compiler- and library-specific.  The language itself doesn't
> specify how it is done.

Thanks, I've tried this but it does NOT work :(

This code prints: 0, it should print: 10

int get_malloc_size(void *vp)
{
  char *cp = (char*)vp;
  static int __done = -1;
  static int __offset = -1;
  if(__done != 0) {
    char *p;
    int n,i;
    while(1) {
      n=random()%10000;
      p=(char*)malloc(n);
      for(i=1;i<100;i++) {
        if(*(int*)(p-i)==n) {
          __offset=i;
          break;
        }
      }
      free(p);
      if(__offset != -1)
        break;
    }
  __done=0;
  }
  return *(int*)(cp-__offset);
}

main()
{
  char *p=(char*)malloc(10);
  printf("%d\n", get_malloc_size((void*)p));
  free(p);
}

[toc] | [prev] | [next] | [standalone]

#17484

In <jhk0oi$8cu$1@speranza.aioe.org> pembed2012 <once@was.enough.invalid> writes:

> Thanks, I've tried this but it does NOT work :(

I suggested one possible way that malloc might store the size of the
allocated area.  There are lots of different ways this can be done.  Your
malloc implementation must use a different mechanism.

I thought I was clear when I said:

> > It's all compiler- and library-specific.  The language itself doesn't
> > specify how it is done.

If you want to know how it's done on your system, you'll have to look
at the system documentation.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon@panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

[toc] | [prev] | [next] | [standalone]

#17485

pembed2012 <once@was.enough.invalid> writes:
> On Thu, 16 Feb 2012 20:48:06 +0000, John Gordon wrote:
>
>> In <jhjofu$jer$1@speranza.aioe.org> pembed2012 <once@was.enough.invalid>
>> writes:
[...]
>>> How does free know how many bytes to free? I have heard that malloc
>>> also allocate additional memory to hold some header information. Where
>>> is this header information stored? If we have access to the header, we
>>> can find that out?
>> 
>> It's all compiler- and library-specific.  The language itself doesn't
>> specify how it is done.
>
> Thanks, I've tried this but it does NOT work :(
>
> This code prints: 0, it should print: 10
>
> int get_malloc_size(void *vp)
> {
>   char *cp = (char*)vp;
>   static int __done = -1;
>   static int __offset = -1;

Identifiers starting with two underscores are reserved to the
implementation.  You should *never* define such identifiers in
your own code.  The rules are a little more complex for identifiers
starting with a single underscore but you should still avoid them.

[snip]

>   return *(int*)(cp-__offset);
> }
>
> main()

Should be "int main(void)".

> {
>   char *p=(char*)malloc(10);
>   printf("%d\n", get_malloc_size((void*)p));
>   free(p);
> }

Why would you expect it to work?  There is *no* portable way to
determine the size of a malloc()ed chunk of memory.

Actually, there is: just remember the argument you passed to malloc():

    size_t requested_size = 10;
    char *p = malloc(requested_size);
    if (p != NULL) {
        /* You know you've allocated requested_size bytes. */
    }

Note that the implementation could allocate more memory than you
request; it won't necessarily store the requested size anywhere.
But that shouldn't matter to your program.

If you think it does matter, explain what you're trying to accomplish
and we can probably help you find another way.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
    Will write code for food.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#17530

On Thu, 16 Feb 2012 14:50:13 -0800, Keith Thompson wrote:

> pembed2012 <once@was.enough.invalid> writes:
>> On Thu, 16 Feb 2012 20:48:06 +0000, John Gordon wrote:
>>
>>> In <jhjofu$jer$1@speranza.aioe.org> pembed2012
>>> <once@was.enough.invalid> writes:
> [...]
>>>> How does free know how many bytes to free? I have heard that malloc
>>>> also allocate additional memory to hold some header information.
>>>> Where is this header information stored? If we have access to the
>>>> header, we can find that out?
>>> 
>>> It's all compiler- and library-specific.  The language itself doesn't
>>> specify how it is done.
>>
>> Thanks, I've tried this but it does NOT work :(
>>
>> This code prints: 0, it should print: 10
>>
>> int get_malloc_size(void *vp)
>> {
>>   char *cp = (char*)vp;
>>   static int __done = -1;
>>   static int __offset = -1;
> 
> Identifiers starting with two underscores are reserved to the
> implementation.  You should *never* define such identifiers in your own
> code.  The rules are a little more complex for identifiers starting with
> a single underscore but you should still avoid them.
> 
> [snip]
> 
>>   return *(int*)(cp-__offset);
>> }
>>
>> main()
> 
> Should be "int main(void)".
> 
>> {
>>   char *p=(char*)malloc(10);
>>   printf("%d\n", get_malloc_size((void*)p)); free(p);
>> }
> 
> Why would you expect it to work?  There is *no* portable way to
> determine the size of a malloc()ed chunk of memory.
> 
> Actually, there is: just remember the argument you passed to malloc():
> 
>     size_t requested_size = 10;
>     char *p = malloc(requested_size);
>     if (p != NULL) {
>         /* You know you've allocated requested_size bytes. */
>     }
> 
> Note that the implementation could allocate more memory than you
> request; it won't necessarily store the requested size anywhere. But
> that shouldn't matter to your program.
> 
> If you think it does matter, explain what you're trying to accomplish
> and we can probably help you find another way.

Sorry it wasn't clear, I need to take a pointer from malloc and find out 
the size of the allocation. To allow for dynamic pointers as well as 
stack arrays. Like this:

myfunc(void* buf){
  int bufsiz=-1;
  if(sizeof(buf)==sizeof(void*)) { // pointer
    bufsiz=get_malloc_size(buf)'
  }
  if(bufsize==-1){ // array
    bufsiz=sizeof(buf);
  }
  // .... process buffer ...
}

Fully portable is best, but a solution working on 90% of compilers would 
be OK if I understood right for you say it can't be done 100%

[toc] | [prev] | [next] | [standalone]

#17531

In article <jhmaol$qn4$1@speranza.aioe.org>,
 pembed2012 <once@was.enough.invalid> wrote:

> Fully portable is best, but a solution working on 90% of compilers would 
> be OK if I understood right for you say it can't be done 100%

A fully portable solution is to include the buffer size packaged with the buffer 
contents, perhaps as a single struct like
        struct buffer {size_t size; Type content[1];}
You are allowed to cast a pointer to any struct
        struct {size_t size; Type content[n];}
to (struct buffer*) and access to p->content[k] for 0<=k<n.

-- 
My name Indigo Montoya.      | What do we do with a drunken sailor?
You flamed my father.        |       I'm whoever you want me to be.
Prepare to be spanked.       |  Annoying Usenet one post at a time.
Stop posting that!           |    At least I can stay in character.

[toc] | [prev] | [next] | [standalone]

#17532

On 2012-02-17, pembed2012 <once@was.enough.invalid> wrote:
> myfunc(void* buf){
>   int bufsiz=-1;
>   if(sizeof(buf)==sizeof(void*)) { // pointer

Since the type of buf is void*, the condition is always true.

>     bufsiz=get_malloc_size(buf)'
>   }
>   if(bufsize==-1){ // array
>     bufsiz=sizeof(buf);
>   }
>   // .... process buffer ...
> }
>
> Fully portable is best, but a solution working on 90% of compilers would 
> be OK if I understood right for you say it can't be done 100%

One method that works 100% is to remember the size of the buffer
in a variable, and pass that information to myfunc along with the
pointer to the buffer.

  #include <stdlib.h>

  void myfunc(void *buf, size_t bufsize);

  void using_pointer(void)
  {
    size_t size = 684;
    char *pointer = malloc(size);
    if (pointer != NULL)
    {
      myfunc(pointer, size);
    }
  }

  void using_array(void)
  {
    char array[684];
    myfunc(array, sizeof array);
  }

[toc] | [prev] | [next] | [standalone]

#17534

On 02/17/2012 02:45 PM, pembed2012 wrote:
...
> Sorry it wasn't clear, I need to take a pointer from malloc and find out 
> the size of the allocation. To allow for dynamic pointers as well as 
> stack arrays. Like this:
> 
> myfunc(void* buf){
>   int bufsiz=-1;

It's absolutely guaranteed, at this point, that sizeof(buf) ==
sizeof(void*). As a result, 2 of your next three lines are pointless,
which suggests that you were not aware of that fact.

>   if(sizeof(buf)==sizeof(void*)) { // pointer
>     bufsiz=get_malloc_size(buf)'
>   }

Why did you think it was possible for the condition of that if() to not
be met?
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#17538

On Fri, 17 Feb 2012 15:33:58 -0500, James Kuyper wrote:
> On 02/17/2012 02:45 PM, pembed2012 wrote: ...
>> Sorry it wasn't clear, I need to take a pointer from malloc and find
>> out the size of the allocation. To allow for dynamic pointers as well
>> as stack arrays. Like this:
>> 
>> myfunc(void* buf){
>>   int bufsiz=-1;
> 
> It's absolutely guaranteed, at this point, that sizeof(buf) ==
> sizeof(void*). As a result, 2 of your next three lines are pointless,
> which suggests that you were not aware of that fact.
> 
>>   if(sizeof(buf)==sizeof(void*)) { // pointer
>>     bufsiz=get_malloc_size(buf)'
>>   }
> 
> Why did you think it was possible for the condition of that if() to not
> be met?

Often this function will be called with a dynamic pointer from malloc. 
However, it is also possible to call it with a stack array:

char ary[10];
char* ptr=malloc(10);
myfunc((void*)ary);  // should work
myfunc((void*)ptr);  // should also work

It started off with the buffers being NULL-free strings. However now I 
need to work with strings with embedded NULL characters. This means I 
can't find the length of the buffer with strlen. And it would be too much 
change to add a new argument to pass the length of the buffer. So the 
only good solution is for myfunc to work out by it's self 1) if the 
buffer is a dynamic pointer or stack array and 2) the size of the buffer.

[toc] | [prev] | [next] | [standalone]

#17539

On 02/17/2012 05:13 PM, pembed2012 wrote:
> On Fri, 17 Feb 2012 15:33:58 -0500, James Kuyper wrote:
>> On 02/17/2012 02:45 PM, pembed2012 wrote: ...
>>> Sorry it wasn't clear, I need to take a pointer from malloc and find
>>> out the size of the allocation. To allow for dynamic pointers as well
>>> as stack arrays. Like this:
>>>
>>> myfunc(void* buf){
>>>   int bufsiz=-1;
>>
>> It's absolutely guaranteed, at this point, that sizeof(buf) ==
>> sizeof(void*). As a result, 2 of your next three lines are pointless,
>> which suggests that you were not aware of that fact.
>>
>>>   if(sizeof(buf)==sizeof(void*)) { // pointer
>>>     bufsiz=get_malloc_size(buf)'
>>>   }
>>
>> Why did you think it was possible for the condition of that if() to not
>> be met?
> 
> Often this function will be called with a dynamic pointer from malloc. 
> However, it is also possible to call it with a stack array:
> 
> char ary[10];
> char* ptr=malloc(10);
> myfunc((void*)ary);  // should work
> myfunc((void*)ptr);  // should also work

Assuming that the function's declaration matches the definition you've
given above, the casts are unnecessary in your code above; they specify
the same conversion that will happen anyway even if they weren't present.

Experienced programmers treat every occurrence of a cast as a potential
time bomb that needs to be examined carefully to make sure it's not an
actual bomb. While your code has mistakes, that's not one of them, so
you're better off removing those unnecessary casts.

Again, why do you think sizeof(buf)==sizeof(void*) will be false for one
of those two cases (presumably the first one)?

Incidentally, it's the wrong test even if it could work the way you were
expecting it to do. Consider:

	char arr[sizeof(void*)];

> It started off with the buffers being NULL-free strings. However now I 
> need to work with strings with embedded NULL characters. This means I 
> can't find the length of the buffer with strlen. And it would be too much 
> change to add a new argument to pass the length of the buffer. So the 
> only good solution is for myfunc to work out by it's self 1) if the 
> buffer is a dynamic pointer or stack array and 2) the size of the buffer.

Unfortunately, it's not possible for your function to figure out either
of those things in the way that you want it to. Your best solution is to
make the changes needed to add a new argument containing the length of
the buffer.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#17543

On 2/17/2012 17:13, pembed2012 wrote:
> On Fri, 17 Feb 2012 15:33:58 -0500, James Kuyper wrote:
>> On 02/17/2012 02:45 PM, pembed2012 wrote: ...
>>> Sorry it wasn't clear, I need to take a pointer from malloc and find
>>> out the size of the allocation. To allow for dynamic pointers as well
>>> as stack arrays. Like this:
>>>
>>> myfunc(void* buf){
>>>    int bufsiz=-1;
>>
>> It's absolutely guaranteed, at this point, that sizeof(buf) ==
>> sizeof(void*). As a result, 2 of your next three lines are pointless,
>> which suggests that you were not aware of that fact.
>>
>>>    if(sizeof(buf)==sizeof(void*)) { // pointer
>>>      bufsiz=get_malloc_size(buf)'
>>>    }
>>
>> Why did you think it was possible for the condition of that if() to not
>> be met?
>
> Often this function will be called with a dynamic pointer from malloc.
> However, it is also possible to call it with a stack array:
>
> char ary[10];
> char* ptr=malloc(10);
> myfunc((void*)ary);  // should work
> myfunc((void*)ptr);  // should also work
>
> It started off with the buffers being NULL-free strings. However now I
> need to work with strings with embedded NULL characters. This means I
> can't find the length of the buffer with strlen. And it would be too much
> change to add a new argument to pass the length of the buffer. So the
> only good solution is for myfunc to work out by it's self 1) if the
> buffer is a dynamic pointer or stack array and 2) the size of the buffer.

I don't think a string has 'NULL' in it.  I think a string can have a 
null terminator.  I think that 'NULL' is an invalid pointer value.

I think you are mistaken about what you are doing and expecting.

It looks like 'ary' has type 'char[10]'.  When 'ary' is used in your 
examples, it yields a 'char *' value.  All pointer-to-object types can 
be implicitly converted to 'void *', so your cast is redundant.

It looks like the type of 'ptr' is 'char *'.  All pointer-to-object 
types can be implicitly converted to 'void *', so your cast is redundant.

In C, if you have a pointer into some object (single object, array 
object, "stack," allocated, static-duration; no difference), I do not 
believe that you can expect that a pointer value carries size 
information nor that even if it does, that you can access that information.

[toc] | [prev] | [next] | [standalone]

#17545

pembed2012 <once@was.enough.invalid> writes:
[...]
> Often this function will be called with a dynamic pointer from malloc. 
> However, it is also possible to call it with a stack array:
>
> char ary[10];
> char* ptr=malloc(10);
> myfunc((void*)ary);  // should work
> myfunc((void*)ptr);  // should also work

Not possible.  It's not just not 100% portable, it's not possible in any
implementation I know of.  You'll need to find a way to keep track of
the size yourself.

> It started off with the buffers being NULL-free strings. However now I 
> need to work with strings with embedded NULL characters.

NULL is (a macro that expands to) a null *pointer* constant.  You're
talking about null *characters*.  I know that some character set
standards use the term "NULL", but you shouldn't use that term to refer
to null characters in the context of C.

A "string" in C is, by definition, "a contiguous sequence of characters
terminated by and including the first null character", so there's no
such thing as a "null-free string".  A character array that doesn't
contain any null characters is not a string; it's simply a character
array.

>                                                          This means I 
> can't find the length of the buffer with strlen.

strlen doesn't give you the size of a buffer; it searches for the
first null character.  You need to decide whether you want the
length of a string contained in a buffer, or the size of the buffer
itself, and you can't reasonably decide that based on its content.
If the buffer holds arbitrary data, it might still contain a null
character that's not meant to be a string terminator; what should
your function return in that case, and how can it decide?

>                                                  And it would be too much 
> change to add a new argument to pass the length of the buffer. So the 
> only good solution is for myfunc to work out by it's self 1) if the 
> buffer is a dynamic pointer or stack array and 2) the size of the buffer.

Then I'm afraid you've got "too much change" ahead of you -- unless
perhaps you can reformulate the problem somehow.  (We don't know *why*
you need this functionality.)

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
    Will write code for food.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#17546

On 2/17/2012 18:01, Keith Thompson wrote:
>>                                                   And it would be too much
>> change to add a new argument to pass the length of the buffer. So the
>> only good solution is for myfunc to work out by it's self 1) if the
>> buffer is a dynamic pointer or stack array and 2) the size of the buffer.
>
> Then I'm afraid you've got "too much change" ahead of you -- unless
> perhaps you can reformulate the problem somehow.  (We don't know *why*
> you need this functionality.)
>

Or the OP could wrap the *alloc functions and 'free' so that the size is 
tracked.  They could even #define over the *alloc functions and 'free'. 
  Then they could also implement their 'alloc_size' function or whatever 
it was called.

Of course, doing that doesn't help with the misunderstanding that the 
size of an _array_ is somehow available from a pointer _value_.

[toc] | [prev] | [next] | [standalone]

#17549

On 02/17/2012 04:13 PM, pembed2012 wrote:
> It started off with the buffers being NULL-free strings. However now I 
> need to work with strings with embedded NULL characters. This means I 
> can't find the length of the buffer with strlen. And it would be too much 
> change to add a new argument to pass the length of the buffer. So the 
> only good solution is for myfunc to work out by it's self 1) if the 
> buffer is a dynamic pointer or stack array and 2) the size of the buffer.

Prepare to be disappointed.

-- 
========================================================================
Ian Pilcher                                         arequipeno@gmail.com
"If you're going to shift my paradigm ... at least buy me dinner first."
========================================================================

[toc] | [prev] | [next] | [standalone]

#17562

"pembed2012" <once@was.enough.invalid> wrote in message
news:jhmjds$is4$1@speranza.aioe.org...


> It started off with the buffers being NULL-free strings. However now I
> need to work with strings with embedded NULL characters. This means I
> can't find the length of the buffer with strlen. And it would be too much
> change to add a new argument to pass the length of the buffer. So the
> only good solution is for myfunc to work out by it's self 1) if the
> buffer is a dynamic pointer or stack array and 2) the size of the buffer.

It is just nul-characters inside the strings, or could it be any char value?

If they are normal strings but with a zero-char included, then just use
another value to use as a terminator, with a special version of strlen().
(Such strings won't work with most of the standard string functions anyway.)

If all the char values could appear in the string (eg. codes 0 to 255 for an
8-bit char), then you can't use a terminator value.

But this is no different from knowing the size of a dynamically allocated
array.

If it's not possible to supply a separate length parameter, then there are
any number of ways to encode the length in the data. As 1, 2 or 4 bytes at
the start, or just before the start, of the string for example. Or perhaps 
use an int-array instead, then you can use the terminator. Or instead pass a 
pointer to a (char*, int) struct where the string and length are passed that 
way. Or... it's difficult to know what's viable and what's isn't, or would 
be too much work.

You can't reliably get the info you need from malloc() anyway.

-- 
Bartc



 

[toc] | [prev] | [next] | [standalone]

#17563

On Feb 17, 5:13 pm, pembed2012 <o...@was.enough.invalid> wrote:
> On Fri, 17 Feb 2012 15:33:58 -0500, James Kuyper wrote:
> > On 02/17/2012 02:45 PM, pembed2012 wrote: ...
> >> Sorry it wasn't clear, I need to take a pointer from malloc and find
> >> out the size of the allocation. To allow for dynamic pointers as well
> >> as stack arrays. Like this:
>
> >> myfunc(void* buf){
> >>   int bufsiz=-1;
>
> > It's absolutely guaranteed, at this point, that sizeof(buf) ==
> > sizeof(void*). As a result, 2 of your next three lines are pointless,
> > which suggests that you were not aware of that fact.
>
> >>   if(sizeof(buf)==sizeof(void*)) { // pointer
> >>     bufsiz=get_malloc_size(buf)'
> >>   }
>
> > Why did you think it was possible for the condition of that if() to not
> > be met?
>
> Often this function will be called with a dynamic pointer from malloc.
> However, it is also possible to call it with a stack array:
>
> char ary[10];
> char* ptr=malloc(10);
> myfunc((void*)ary);  // should work
> myfunc((void*)ptr);  // should also work
>
> It started off with the buffers being NULL-free strings. However now I
> need to work with strings with embedded NULL characters. This means I
> can't find the length of the buffer with strlen. And it would be too much
> change to add a new argument to pass the length of the buffer. So the
> only good solution is for myfunc to work out by it's self 1) if the
> buffer is a dynamic pointer or stack array and 2) the size of the buffer.

Since you seem quite interested in deriving how to access the size of
an allocation from just a pointer, one method is to store the size
yourself in your own allocator that wraps 'malloc'.

\code snippet
uintmax_t current_memory;

void* track_malloc( size_t size )
{
  void* mem_blk = NULL;
  void* p = NULL;

  /*
   * To track the amount of used memory, each memory allocation is
   * prefixed with a size_t object that allows the free function to
   * update 'current_memory' correctly when releasing memory.
   */
  mem_blk = malloc( sizeof (size_t) + size );

  if ( mem_blk )
  {
    *((size_t*)mem_blk) = size;
    p = (unsigned char*)mem_blk + sizeof (size_t);

    current_memory += size;
  }

  return p;
}

void track_free( void* p )
{
  void* mem_blk = NULL;
  size_t p_size;

  if ( p )
  {
    mem_blk = (unsigned char*)p - sizeof (size_t);
    p_size = *((size_t*)mem_blk);

    free( mem_blk );

    current_memory -= p_size;
  }
}
\endcode

In essence, it adds a header to each pointer that stores the size of
the allocation in a 'size_t' prefix.

      -------- ----------------------------
     | size_t |       ... object ...       |
      -------- ----------------------------
     /        /
  mem        p

Everything seems fine until you decide to allocate an object that has
stricter alignment requirements than a 'size_t'.

      -------- --------------
     | size_t |    double    | (xxx not aligned xxx)
      -------- --------------
     /        /
  mem        p

One of the requirements for malloc is to provide a memory block that
is aligned for any object.  So while 'mem' is aligned properly for a
double, the double object at 'p' is not.  For example, on my 32-bit
Windows, 'size_t' has an alignment of 4, while 'double' has an
alignment of 8.  The proper method to prefix the allocation pointer
for a 'double' is to insert a 'double' sized object.

         4         4          8
      -------- -------- --------------
     | size_t |   pad  |    double    |
      -------- -------- --------------
     /                 /
  mem                 p

Now your 'double' object is guaranteed to be aligned.  The general
solution is to prefix each memory allocation with the maximum
alignment detected from the system, referenced as ALIGN_MAX.  This can
be defined by the compiler (under the newest C standard), or estimated
using offsetof and a union of a large set of distinct types.  Search
for Chris Thomasson who has posted about it before.

So instead of using 'sizeof' to calculate the size of the 'size_t'
prefix, you'll need a different version to pad it to a multiple of
ALIGN_MAX.

\code snippet
uintmax_t current_memory;

size_t gc_maxalign_sizeof( size_t type_sz )
{ return ((type_sz + ALIGN_MAX - 1) / ALIGN_MAX) * ALIGN_MAX; }

#define c_maxalign_sizeof( type ) ( (gc_maxalign_sizeof( sizeof
(type) )) )

void* track_malloc( size_t size )
{
  void* mem_blk = NULL;
  void* p = NULL;

  /*
   * To track the amount of used memory, each memory allocation is
   * prefixed with a size_t object that allows the free function to
   * update 'current_memory' correctly when releasing memory.
   */
  mem_blk = malloc( c_maxalign_sizeof (size_t) + size );

  if ( mem_blk )
  {
    *((size_t*)mem_blk) = size;
    p = (unsigned char*)mem_blk + c_maxalign_sizeof (size_t);

    current_memory += size;
  }

  return p;
}

void track_free( void* p )
{
  void* mem_blk = NULL;
  size_t p_size;

  if ( p )
  {
    mem_blk = (unsigned char*)p - c_maxalign_sizeof (size_t);
    p_size = *((size_t*)mem_blk);

    free( mem_blk );

    current_memory -= p_size;
  }
}
\endcode

Note that while this prefixes the size, none of the string functions
(strlen) know about this size prefix, so you still have to define your
own interface.  It does allow you to get away with passing one
augmented pointer since you pack the allocation size with it.

size_t track_strlen( const char* str )
{
  void* mem_blk = (unsigned char*)str - c_maxalign_sizeof (size_t);
  return *((size_t*)mem_blk);
}

I'd consider using a special typedef to distinguish these augmented
string pointers to distinguish functions that expect them and those
that do not.

This is the just one possible path.  Another path is to define a
string structure that stores the length and buffer size.

struct c_string
{
  char* buf;
  size_t buf_size;
  size_t length;
};

Then you write a string interface that works with 'struct c_string',
which is able to handle embedded nul characters since you store the
length.

\code snippet
size_t c_string_length( const struct c_string* cstr )
{
  return cstr->length;
}
\endcode

You can write your own (a lot of time/work), or decide to use one of
the string libraries out there, like bstring.  I would strongly
consider using an external string library was created to handle
strings with embedded nulls.  If you can walk on someone else's
shoulders to step over a mud pit ...

Best regards,
John D.

[toc] | [prev] | [next] | [standalone]

#17564

On 2012-02-17, pembed2012 <once@was.enough.invalid> wrote:
> It started off with the buffers being NULL-free strings. However now I 
> need to work with strings with embedded NULL characters. This means I 

So the requirements changed.

> can't find the length of the buffer with strlen. And it would be too much 
> change to add a new argument to pass the length of the buffer. 

That is the right thing to do.  Guess how long it will take you to do it
and then give the estimate to whoever changed the requirement.
Aren't you being paid?

> So the 
> only good solution is for myfunc to work out by it's self 1) if the 
> buffer is a dynamic pointer or stack array and 2) the size of the buffer.

This is not possible in any portable way. Some malloc implementations have a
function which can tell you how large the underlying block is (but not
necessarily with byte accuracy). As far as obtaining the size of an array in
automatic storage, forget it.

You can add a length field to string objects without introducing a new
parameter everywhere. Just change the type from "char *" to "string *",
where string is a struct type such as:

  typedef struct { int size; char *data; } string;

Or even:

  /* flexible array hack */
  typedef struct { int size; char [1]; } string;

[toc] | [prev] | [next] | [standalone]

#17541

On 2/17/2012 14:45, pembed2012 wrote:
>
> Sorry it wasn't clear, I need to take a pointer from malloc and find out
> the size of the allocation. To allow for dynamic pointers as well as
> stack arrays. Like this:
>
> myfunc(void* buf){
>    int bufsiz=-1;
      /* The following condition is always true */
>    if(sizeof(buf)==sizeof(void*)) { // pointer
        /* Oops. Semi-colon needed instead of ' */
>      bufsiz=get_malloc_size(buf)'
>    }
      /* Oops. 'bufsiz', not 'bufsize' */
>    if(bufsize==-1){ // array
        /* This will never happen */
>      bufsiz=sizeof(buf);
>    }
>    // .... process buffer ...
> }
>
> Fully portable is best, but a solution working on 90% of compilers would
> be OK if I understood right for you say it can't be done 100%

   I don't think it matters what kind of pointer-to-object type you 
originally pass to 'myfunc'.  'buf' is a 'void *' and 'sizeof buf' 
should be equal to 'sizeof (void *)'.

Did you understand my example allocator that I posted?  It ought to be 
at least 90% portable.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.c

csiph-web