Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #32281 > unrolled thread

ctypes free memory which is allocated in C DLL

Back to article view | Back to comp.lang.python

Contents

  ctypes free memory which is allocated in C DLL zlchen.ken@gmail.com - 2012-10-27 07:42 -0700
    Re: ctypes free memory which is allocated in C DLL Chris Angelico <rosuav@gmail.com> - 2012-10-28 01:56 +1100
      Re: ctypes free memory which is allocated in C DLL Ken Chen <zlchen.ken@gmail.com> - 2012-10-27 08:40 -0700
        Re: ctypes free memory which is allocated in C DLL Chris Angelico <rosuav@gmail.com> - 2012-10-28 07:26 +1100
      Re: ctypes free memory which is allocated in C DLL Ken Chen <zlchen.ken@gmail.com> - 2012-10-27 08:40 -0700
    Re: ctypes free memory which is allocated in C DLL Nobody <nobody@nowhere.com> - 2012-10-27 23:26 +0100
      Re: ctypes free memory which is allocated in C DLL zlchen.ken@gmail.com - 2012-10-27 19:05 -0700

#32281 — ctypes free memory which is allocated in C DLL

Hi Guys,

I have a DLL which written in C language, one of the function is to allocate a structure, fill the members and then return the pointer of the structure.

After Python called this function, and done with the returned structure, I would like to free the returned structure. How can I achieve this ? Basically, I tried that I wrote a corresponding free interface in the DLL, it works, but calling the libc.free in Python doesn't work.

my_dll.return_structure_ptr.restypes = POINTER(Dummy)
res_ptr = my_dll.return_structure_ptr()
windll.msvcrt.free(res_ptr)  <==== doesn't work, memory violation 
my_dll.free_dummy_struture(res_ptr)  <== This works.

Thanks !

[toc] | [next] | [standalone]

#32282

On Sun, Oct 28, 2012 at 1:42 AM,  <zlchen.ken@gmail.com> wrote:
> Hi Guys,
>
> I have a DLL which written in C language, one of the function is to allocate a structure, fill the members and then return the pointer of the structure.
>
> After Python called this function, and done with the returned structure, I would like to free the returned structure. How can I achieve this ? Basically, I tried that I wrote a corresponding free interface in the DLL, it works, but calling the libc.free in Python doesn't work.

As a general rule, you should always match your allocs and frees. Use
the same library to free the memory as was used to allocate it.
Nothing else is safe. If your DLL exposes an API that allocates
memory, your DLL should expose a corresponding API to free it. So what
you have is the best way :)

ChrisA

[toc] | [prev] | [next] | [standalone]

#32283

On Saturday, October 27, 2012 10:56:54 PM UTC+8, Chris Angelico wrote:
> On Sun, Oct 28, 2012 at 1:42 AM,  <zlchen.ken@gmail.com> wrote:
> 
> > Hi Guys,
> 
> >
> 
> > I have a DLL which written in C language, one of the function is to allocate a structure, fill the members and then return the pointer of the structure.
> 
> >
> 
> > After Python called this function, and done with the returned structure, I would like to free the returned structure. How can I achieve this ? Basically, I tried that I wrote a corresponding free interface in the DLL, it works, but calling the libc.free in Python doesn't work.
> 
> 
> 
> As a general rule, you should always match your allocs and frees. Use
> 
> the same library to free the memory as was used to allocate it.
> 
> Nothing else is safe. If your DLL exposes an API that allocates
> 
> memory, your DLL should expose a corresponding API to free it. So what
> 
> you have is the best way :)
> 
> 
> 
> ChrisA

Thanks a lot, Chris!
Yes, I agree writing a corresponding API to free the memory is the best practice and best bet.
Sometimes, the third party API may not provide that.

After digging the Python manual again and again.
I finally figure out why windll.msvcrt.free is failing.

As the manual stated below, the DLL is using another version of msvcrt lib which is different than the builtin windll.msvcrt. After I explicitly load the msvcrt which built the DLL, things are getting function now.

"ctypes.util.find_msvcrt() 
Windows only: return the filename of the VC runtype library used by Python, and by the extension modules. If the name of the library cannot be determined, None is returned.

If you need to free memory, for example, allocated by an extension module with a call to the free(void *), it is important that you use the function in the same library that allocated the memory."

[toc] | [prev] | [next] | [standalone]

#32290

On Sun, Oct 28, 2012 at 2:40 AM, Ken Chen <zlchen.ken@gmail.com> wrote:
> Yes, I agree writing a corresponding API to free the memory is the best practice and best bet.
> Sometimes, the third party API may not provide that.

Then that's a majorly dangerous third party API. The only time it's
safe to provide a half-only API like that is when the code gets
statically linked with the application, so there's a guarantee that
malloc() in one place corresponds to free() in another.

> After digging the Python manual again and again.
> I finally figure out why windll.msvcrt.free is failing.
>
> As the manual stated below, the DLL is using another version of msvcrt lib which is different than the builtin windll.msvcrt. After I explicitly load the msvcrt which built the DLL, things are getting function now.

That's still vulnerable. There's really no guarantee about _anything_
with mismatched memory allocators; it's theoretically possible for
compiler switches to change the behaviour of malloc/free such that
compiling malloc in debug mode and free in release mode would crash
your program. (I don't know off-hand of any compilers/libraries that
do that specifically, but there are similar changes done in other
ways.)

If a DLL allocates memory and doesn't deallocate it, lean on its
author to complete the job.

ChrisA

[toc] | [prev] | [next] | [standalone]

#32284

On Saturday, October 27, 2012 10:56:54 PM UTC+8, Chris Angelico wrote:
> On Sun, Oct 28, 2012 at 1:42 AM,  <zlchen.ken@gmail.com> wrote:
> 
> > Hi Guys,
> 
> >
> 
> > I have a DLL which written in C language, one of the function is to allocate a structure, fill the members and then return the pointer of the structure.
> 
> >
> 
> > After Python called this function, and done with the returned structure, I would like to free the returned structure. How can I achieve this ? Basically, I tried that I wrote a corresponding free interface in the DLL, it works, but calling the libc.free in Python doesn't work.
> 
> 
> 
> As a general rule, you should always match your allocs and frees. Use
> 
> the same library to free the memory as was used to allocate it.
> 
> Nothing else is safe. If your DLL exposes an API that allocates
> 
> memory, your DLL should expose a corresponding API to free it. So what
> 
> you have is the best way :)
> 
> 
> 
> ChrisA

Thanks a lot, Chris!
Yes, I agree writing a corresponding API to free the memory is the best practice and best bet.
Sometimes, the third party API may not provide that.

After digging the Python manual again and again.
I finally figure out why windll.msvcrt.free is failing.

As the manual stated below, the DLL is using another version of msvcrt lib which is different than the builtin windll.msvcrt. After I explicitly load the msvcrt which built the DLL, things are getting function now.

"ctypes.util.find_msvcrt() 
Windows only: return the filename of the VC runtype library used by Python, and by the extension modules. If the name of the library cannot be determined, None is returned.

If you need to free memory, for example, allocated by an extension module with a call to the free(void *), it is important that you use the function in the same library that allocated the memory."

[toc] | [prev] | [next] | [standalone]

#32294

On Sat, 27 Oct 2012 07:42:01 -0700, zlchen.ken wrote:

> I have a DLL which written in C language, one of the function is to
> allocate a structure, fill the members and then return the pointer of
> the structure. 
> 
> After Python called this function, and done with the returned structure,
> I would like to free the returned structure. How can I achieve this ?
> Basically, I tried that I wrote a corresponding free interface in the
> DLL, it works, but calling the libc.free in Python doesn't work.
> 
> my_dll.return_structure_ptr.restypes = POINTER(Dummy) res_ptr =
> my_dll.return_structure_ptr() windll.msvcrt.free(res_ptr)  <==== doesn't
> work, memory violation my_dll.free_dummy_struture(res_ptr)  <== This
> works.

On Windows, a process may have multiple versions of the MSVCRT DLL (which
provides malloc/free). If an executable or DLL is linked against multiple
DLLs, each DLL could be using a different version of MSVCRT.

Different versions of MSVCRT may have separate heaps, so anything which
is allocated with malloc() (or calloc() or realloc()) from a specific
version of MSVCRT must be passed to free() from the same version of MSVCRT.
windll.msvcrt refers to the version of MSVCRT against which the Python DLL
is linked, which isn't necessarily the version against which my_dll is
linked.

If a function in a DLL returns a pointer to memory which it allocated
with malloc(), that DLL must also provide a function which can be used to
free that memory. It can't leave it to the application (or higher-level
DLL) to call free(), because the application may not be using the same
version of MSVCRT as the DLL.

[toc] | [prev] | [next] | [standalone]

#32298

On Sunday, October 28, 2012 6:26:28 AM UTC+8, Nobody wrote:
> On Sat, 27 Oct 2012 07:42:01 -0700, zlchen.ken wrote:
> 
> 
> 
> > I have a DLL which written in C language, one of the function is to
> 
> > allocate a structure, fill the members and then return the pointer of
> 
> > the structure. 
> 
> > 
> 
> > After Python called this function, and done with the returned structure,
> 
> > I would like to free the returned structure. How can I achieve this ?
> 
> > Basically, I tried that I wrote a corresponding free interface in the
> 
> > DLL, it works, but calling the libc.free in Python doesn't work.
> 
> > 
> 
> > my_dll.return_structure_ptr.restypes = POINTER(Dummy) res_ptr =
> 
> > my_dll.return_structure_ptr() windll.msvcrt.free(res_ptr)  <==== doesn't
> 
> > work, memory violation my_dll.free_dummy_struture(res_ptr)  <== This
> 
> > works.
> 
> 
> 
> On Windows, a process may have multiple versions of the MSVCRT DLL (which
> 
> provides malloc/free). If an executable or DLL is linked against multiple
> 
> DLLs, each DLL could be using a different version of MSVCRT.
> 
> 
> 
> Different versions of MSVCRT may have separate heaps, so anything which
> 
> is allocated with malloc() (or calloc() or realloc()) from a specific
> 
> version of MSVCRT must be passed to free() from the same version of MSVCRT.
> 
> windll.msvcrt refers to the version of MSVCRT against which the Python DLL
> 
> is linked, which isn't necessarily the version against which my_dll is
> 
> linked.
> 
> 
> 
> If a function in a DLL returns a pointer to memory which it allocated
> 
> with malloc(), that DLL must also provide a function which can be used to
> 
> free that memory. It can't leave it to the application (or higher-level
> 
> DLL) to call free(), because the application may not be using the same
> 
> version of MSVCRT as the DLL.

Thank you for the details.
This is really useful!

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web