Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #14113 > unrolled thread

malloc

Back to article view | Back to comp.lang.c++

Contents

  malloc jgk@panix.com (Joe keane) - 2012-02-16 21:04 +0000
    Re: malloc Devil with the China Blue Dress <chine.bleu@yahoo.com> - 2012-02-16 13:21 -0800
      Re: malloc BGB <cr88192@hotmail.com> - 2012-02-16 16:16 -0700
        Re: malloc Paavo Helde <myfirstname@osa.pri.ee> - 2012-02-17 14:46 -0600
          Re: malloc Joshua Maurice <joshuamaurice@gmail.com> - 2012-02-17 14:51 -0800
            Re: malloc BGB <cr88192@hotmail.com> - 2012-02-17 19:58 -0700
              Re: malloc Paavo Helde <myfirstname@osa.pri.ee> - 2012-02-18 02:18 -0600
                Re: malloc BGB <cr88192@hotmail.com> - 2012-02-18 01:58 -0700
                Re: malloc glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2012-02-18 15:30 +0000
              Re: malloc Kaz Kylheku <kaz@kylheku.com> - 2012-02-18 10:36 +0000
              Re: malloc William Ahern <william@wilbur.25thandClement.com> - 2012-02-18 16:43 -0800
                Re: malloc BGB <cr88192@hotmail.com> - 2012-02-18 20:03 -0700
                  Re: malloc Miles Bader <miles@gnu.org> - 2012-02-19 12:49 +0900
                    Re: malloc William Ahern <william@wilbur.25thandClement.com> - 2012-02-18 21:08 -0800
                    Re: malloc "io_x" <a@b.c.invalid> - 2012-02-19 18:23 +0100
                  Re: malloc William Ahern <william@wilbur.25thandClement.com> - 2012-02-18 21:06 -0800
                    Re: malloc BGB <cr88192@hotmail.com> - 2012-02-18 23:06 -0700
        Re: malloc "Asger Joergensen" <Junk@Asger-P.dk> - 2012-02-19 15:54 +0000
          Re: malloc BGB <cr88192@hotmail.com> - 2012-02-19 09:37 -0700
      Re: malloc Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-02-16 20:27 -0500
        Re: malloc Goran <goran.pusic@gmail.com> - 2012-02-16 23:40 -0800
          Re: malloc Keith Thompson <kst-u@mib.org> - 2012-02-17 02:26 -0800
            Re: malloc Goran <goran.pusic@gmail.com> - 2012-02-17 04:01 -0800
              Re: malloc Keith Thompson <kst-u@mib.org> - 2012-02-17 09:25 -0800
                Re: malloc 88888 Dihedral <dihedral88888@googlemail.com> - 2012-02-17 09:37 -0800
                Re: malloc Joshua Maurice <joshuamaurice@gmail.com> - 2012-02-17 14:49 -0800
                  Re: malloc glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2012-02-18 01:43 +0000
                    Re: malloc Kaz Kylheku <kaz@kylheku.com> - 2012-02-18 02:04 +0000
                  Re: malloc scott@slp53.sl.home (Scott Lurndal) - 2012-02-18 03:04 +0000
                    Re: malloc Robert Wessel <robertwessel2@yahoo.com> - 2012-02-17 22:15 -0600
                    Re: malloc Joshua Maurice <joshuamaurice@gmail.com> - 2012-02-17 23:48 -0800
                Re: malloc Kaz Kylheku <kaz@kylheku.com> - 2012-02-17 23:29 +0000
          Re: malloc Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-02-17 08:06 -0500
            Re: malloc Goran <goran.pusic@gmail.com> - 2012-02-18 07:57 -0800
              Re: malloc Keith Thompson <kst-u@mib.org> - 2012-02-18 10:57 -0800
              Re: malloc Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-02-19 08:51 -0500
        Re: malloc scott@slp53.sl.home (Scott Lurndal) - 2012-02-17 20:31 +0000
          Re: malloc Joshua Maurice <joshuamaurice@gmail.com> - 2012-02-18 00:01 -0800
            Re: malloc scott@slp53.sl.home (Scott Lurndal) - 2012-02-18 17:07 +0000
              Re: malloc Joshua Maurice <joshuamaurice@gmail.com> - 2012-02-18 18:14 -0800
                Re: malloc scott@slp53.sl.home (Scott Lurndal) - 2012-02-19 17:14 +0000
    Re: malloc Goran <goran.pusic@gmail.com> - 2012-02-16 23:43 -0800
    Re: malloc "io_x" <a@b.c.invalid> - 2012-02-19 16:44 +0100

Page 1 of 3  [1] 2 3  Next page →

#14113 — malloc

Bugs of memory allocation will make you mad.

Bugs *in* memory allocation will put you in the cuckoo people place.

[toc] | [next] | [standalone]

#14117

In article <jhjr14$rjk$1@reader1.panix.com>, jgk@panix.com (Joe keane) wrote:

> Bugs of memory allocation will make you mad.
> 
> Bugs *in* memory allocation will put you in the cuckoo people place.

Which is why they are exceedingly rare. Nearly all allocation problems are due 
to the program storing outside array bounds.

-- 
My name Indigo Montoya.      |         It's Coraline, not Caroline.
You flamed my father.        |       I'm whoever you want me to be.
Prepare to be spanked.       |  Annoying Usenet one post at a time.
Stop posting that!           |    At least I can stay in character.

[toc] | [prev] | [next] | [standalone]

#14122

On 2/16/2012 2:21 PM, Devil with the China Blue Dress wrote:
> In article<jhjr14$rjk$1@reader1.panix.com>, jgk@panix.com (Joe keane) wrote:
>
>> Bugs of memory allocation will make you mad.
>>
>> Bugs *in* memory allocation will put you in the cuckoo people place.
>
> Which is why they are exceedingly rare. Nearly all allocation problems are due
> to the program storing outside array bounds.
>

which are annoyingly difficult to track down sometimes...


it would be nice if compilers would offer an option (as a debugging 
feature) to optionally put in bounds checking for many operations (it is 
also possible to do this without adding fat pointers or fundamentally 
changing how the language works, although it would still introduce a 
run-time cost).

simple idea:
on a array access, identify the memory object pointed-to by the base 
pointer (heap-lookup);
determine if the address being assigned to is the same object (possibly, 
another heap lookup, or checking the target against the first object);
if not, blow up.

now, if the person can't afford the cost, they don't have to use the 
feature.


some other use cases are a little harder, say:
double *fp;
fp=...
while(fp) { *fp=*fp+1; fp++; }

the issue would be that, potentially, the pointer could jump from one 
object to the next without the run-time checks noticing.

a secondary defense could be to place "trip wires" in the heap between 
objects, and if a memory-write check determines that the pointer points 
to such a trip-wire, then an exception can be raised (one option for 
such trip-wires is to have them occupy the same physical space as 
memory-object headers or similar, and maybe also several for words 
following the end of a memory-object).


although, without the explicit testing or throwing (kind of a problem 
with a standard compiler), I have used similar before as a means to 
attempt to debug array overruns with several custom memory managers of 
mine (the memory managers will make some attempt to detect and 
diagnose/report these sorts of problems).


or such...

[toc] | [prev] | [next] | [standalone]

#14194

BGB <cr88192@hotmail.com> wrote in news:jhk2rh$19o$1@news.albasani.net:

> On 2/16/2012 2:21 PM, Devil with the China Blue Dress wrote:
>> In article<jhjr14$rjk$1@reader1.panix.com>, jgk@panix.com (Joe keane)
>> wrote: 
>>
>>> Bugs of memory allocation will make you mad.
>>>
>>> Bugs *in* memory allocation will put you in the cuckoo people place.
>>
>> Which is why they are exceedingly rare. Nearly all allocation
>> problems are due to the program storing outside array bounds.
>>
> 
> which are annoyingly difficult to track down sometimes...
> 
> 
> it would be nice if compilers would offer an option (as a debugging 
> feature) to optionally put in bounds checking for many operations 

Yes it is, and they are doing that. Use e.g. MSVC with iterator checking 
switched on (this is the default) and accessing e.g. a std::vector out of 
bounds will generate a runtime error. With gcc one can use MALLOC_CHECK_ 
or -lmcheck, these also should catch some out-of-bounds access errors.

For raw pointers it is more difficult, you can use something like 
ElectricFence or Valgrind on Linux, but it makes your program to run many 
times slower and consume lots of more memory so this is just for 
debugging.


hth
Paavo

[toc] | [prev] | [next] | [standalone]

#14205

On Feb 17, 12:46 pm, Paavo Helde <myfirstn...@osa.pri.ee> wrote:
> BGB <cr88...@hotmail.com> wrote innews:jhk2rh$19o$1@news.albasani.net:
>
>
>
>
>
>
>
>
>
> > On 2/16/2012 2:21 PM, Devil with the China Blue Dress wrote:
> >> In article<jhjr14$rj...@reader1.panix.com>, j...@panix.com (Joe keane)
> >> wrote:
>
> >>> Bugs of memory allocation will make you mad.
>
> >>> Bugs *in* memory allocation will put you in the cuckoo people place.
>
> >> Which is why they are exceedingly rare. Nearly all allocation
> >> problems are due to the program storing outside array bounds.
>
> > which are annoyingly difficult to track down sometimes...
>
> > it would be nice if compilers would offer an option (as a debugging
> > feature) to optionally put in bounds checking for many operations
>
> Yes it is, and they are doing that. Use e.g. MSVC with iterator checking
> switched on (this is the default) and accessing e.g. a std::vector out of
> bounds will generate a runtime error. With gcc one can use MALLOC_CHECK_
> or -lmcheck, these also should catch some out-of-bounds access errors.
>
> For raw pointers it is more difficult, you can use something like
> ElectricFence or Valgrind on Linux, but it makes your program to run many
> times slower and consume lots of more memory so this is just for
> debugging.

MSVC also has some debug options that try to catch writes past the
ends of allocated regions as well. IIRC, they overallocate, and put
special bit patterns at the start and end on allocation, and when
freed they check to see if those bit patterns are intact, raising a
fatal error or something if it finds a problem.

[toc] | [prev] | [next] | [standalone]

#14217

On 2/17/2012 3:51 PM, Joshua Maurice wrote:
> On Feb 17, 12:46 pm, Paavo Helde<myfirstn...@osa.pri.ee>  wrote:
>> BGB<cr88...@hotmail.com>  wrote innews:jhk2rh$19o$1@news.albasani.net:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> On 2/16/2012 2:21 PM, Devil with the China Blue Dress wrote:
>>>> In article<jhjr14$rj...@reader1.panix.com>, j...@panix.com (Joe keane)
>>>> wrote:
>>
>>>>> Bugs of memory allocation will make you mad.
>>
>>>>> Bugs *in* memory allocation will put you in the cuckoo people place.
>>
>>>> Which is why they are exceedingly rare. Nearly all allocation
>>>> problems are due to the program storing outside array bounds.
>>
>>> which are annoyingly difficult to track down sometimes...
>>
>>> it would be nice if compilers would offer an option (as a debugging
>>> feature) to optionally put in bounds checking for many operations
>>
>> Yes it is, and they are doing that. Use e.g. MSVC with iterator checking
>> switched on (this is the default) and accessing e.g. a std::vector out of
>> bounds will generate a runtime error. With gcc one can use MALLOC_CHECK_
>> or -lmcheck, these also should catch some out-of-bounds access errors.
>>
>> For raw pointers it is more difficult, you can use something like
>> ElectricFence or Valgrind on Linux, but it makes your program to run many
>> times slower and consume lots of more memory so this is just for
>> debugging.
>

I was mildly aware of ElectricFence and Valgrind, and was also thinking 
mostly of malloc'ed raw pointers and C-style arrays (when passed as 
pointers). the idea would be if something like Valgrind were directly 
integrated into the compiler/runtime as a debug option. this is mostly 
what I was writing about.

as for bounds-checked collection types, yes, I have a few of those as 
well, and also mostly use a custom memory manager (mostly for GC and 
dynamic type-checking), which could (sadly) create issues (likely not 
detect bounds violations) if this feature were available.


> MSVC also has some debug options that try to catch writes past the
> ends of allocated regions as well. IIRC, they overallocate, and put
> special bit patterns at the start and end on allocation, and when
> freed they check to see if those bit patterns are intact, raising a
> fatal error or something if it finds a problem.

this is partly what my memory allocators do, but may also under certain 
cases scan the heap, detect corruption, and attempt to diagnose it.

a nifty tool I have used in some places is object-origin tracking, where 
every time the allocator is accessed, it records where it was called 
from, and will use this information (combined with some data-forensics) 
to try to make an educated guess as to "who done it" (or, IOW, around 
where the offending code might be).

although, sadly, there is no good way to implement a HW write barrier 
for this (sadly, neither Windows nor Linux give enough control over the 
CPU to really make something like this all that workable, even then it 
would still likely be page-fault driven slowness).

I have some stuff for "software write barriers" (mostly needed for other 
reasons), but not a lot of code uses them (unless forced into it), 
partly due to the added awkwardness and performance overheads.

one option that could sort of work for larger arrays would be to put 
unused pages between memory objects, such that going out of bounds is 
more likely to trigger a page fault.


or such...

[toc] | [prev] | [next] | [standalone]

#14226

BGB <cr88192@hotmail.com> wrote in news:jhn46t$dlp$1@news.albasani.net:

> 
> a nifty tool I have used in some places is object-origin tracking,
> where every time the allocator is accessed, it records where it was
> called from, and will use this information (combined with some
> data-forensics) to try to make an educated guess as to "who done it"
> (or, IOW, around where the offending code might be).
> 
> although, sadly, there is no good way to implement a HW write barrier 
> for this (sadly, neither Windows nor Linux give enough control over
> the CPU to really make something like this all that workable, even
> then it would still likely be page-fault driven slowness).

This has been done in hardware as well. 
http://en.wikipedia.org/wiki/Bounds_checking mentions VAX, B6500 and 
Burroughs. By some reason this approach has not really catched on.

> 
> one option that could sort of work for larger arrays would be to put 
> unused pages between memory objects, such that going out of bounds is 
> more likely to trigger a page fault.

This is exactly what Electric Fence does. It is also slow as hell, I 
guess the most slowdown comes from where the array ends in a middle of a 
virtual memory page and efence has to decide after a page fault if the 
access was legal or not. For small allocations it also produces severe 
memory fragmentation. 

If you are only interested in arrays, then some compiler support would be 
indeed helpful. Electric Fence intercepts all malloc calls and does not 
know if these are meant for arrays or not (I guess), so it has to protect 
all of them.

However, I would say that the best advice for avoiding out-of-bounds 
access errors with raw pointers in C++ is to not use raw pointers.

Cheers
Paavo

[toc] | [prev] | [next] | [standalone]

#14227

On 2/18/2012 1:18 AM, Paavo Helde wrote:
> BGB<cr88192@hotmail.com>  wrote in news:jhn46t$dlp$1@news.albasani.net:
>
>>
>> a nifty tool I have used in some places is object-origin tracking,
>> where every time the allocator is accessed, it records where it was
>> called from, and will use this information (combined with some
>> data-forensics) to try to make an educated guess as to "who done it"
>> (or, IOW, around where the offending code might be).
>>
>> although, sadly, there is no good way to implement a HW write barrier
>> for this (sadly, neither Windows nor Linux give enough control over
>> the CPU to really make something like this all that workable, even
>> then it would still likely be page-fault driven slowness).
>
> This has been done in hardware as well.
> http://en.wikipedia.org/wiki/Bounds_checking mentions VAX, B6500 and
> Burroughs. By some reason this approach has not really catched on.
>

not nearly so helpful on the PC though, where one is limited to options 
both available for x86 and also that will work directly on OS's like 
Windows and Linux.

>>
>> one option that could sort of work for larger arrays would be to put
>> unused pages between memory objects, such that going out of bounds is
>> more likely to trigger a page fault.
>
> This is exactly what Electric Fence does. It is also slow as hell, I
> guess the most slowdown comes from where the array ends in a middle of a
> virtual memory page and efence has to decide after a page fault if the
> access was legal or not. For small allocations it also produces severe
> memory fragmentation.
>

yeah. there are ways it could be done a little faster, say if one had 
access to ring-2 or similar. if one is stuck with ring-3, then one has 
to jerk off with the page access every time one wants to change 
something (that or use double-mapping). the general problem though is 
that this sort of thing isn't really fast, which is sort of why it would 
be nicer if the compiler could handle it in software (internally 
generating calls to write-barrier functions or similar).


> If you are only interested in arrays, then some compiler support would be
> indeed helpful. Electric Fence intercepts all malloc calls and does not
> know if these are meant for arrays or not (I guess), so it has to protect
> all of them.
>

I had assumed the possibility of checking near all assignments via 
pointers within a given compilation unit if a certain flag were used.

> However, I would say that the best advice for avoiding out-of-bounds
> access errors with raw pointers in C++ is to not use raw pointers.
>

I use both C++ and a lot of plain C as well (in a project currently with 
parts written in 5 different languages, though C is the majority leader).

hence, in many cases raw pointers are often "the normal way of doing 
things" (except when using dynamic types, or dedicated array objects).


or such...

[toc] | [prev] | [next] | [standalone]

#14234

In comp.lang.c++ Paavo Helde <myfirstname@osa.pri.ee> wrote:
(snip)

>> the CPU to really make something like this all that workable, even
>> then it would still likely be page-fault driven slowness).

> This has been done in hardware as well. 
> http://en.wikipedia.org/wiki/Bounds_checking mentions VAX, B6500 and 
> Burroughs. By some reason this approach has not really catched on.

As I remember VAX allows pages to be marked read-only.
I remember Fortran compilers that would put constants in
read-only pages, such that they couldn't accidentally be 
modified. With call-by-reference, modification of constants
has a long history in Fortran.

>> one option that could sort of work for larger arrays would be 
>> to put unused pages between memory objects, such that going 
>> out of bounds is more likely to trigger a page fault.

> This is exactly what Electric Fence does. It is also slow as hell, 
> I guess the most slowdown comes from where the array ends in 
> a middle of a virtual memory page and efence 
(snip)

In the 16 bit OS/2 days, when most people were still using 16-bit
DOS, I would allocate arrays by directly calling the OS/2
segment allocation routine, for a segment exactly the size
required. That length then goes into the hardware segment
descriptor register, such that hardware tests the bounds,
even on fetch. That was before paging, though.
(Eventually you run into the 8K segment descriptor per
process limit, but usually it works pretty well.)

If Intel had added a segment descriptor cache to any of
the processors, it would have greatly speeded up some of the
things that were done in the 16 bit, and early 32 bit, OS/2 days.

If done right, it would have delayed the need for 64 bit
processors by some years.

-- glen

[toc] | [prev] | [next] | [standalone]

#14230

On 2012-02-18, BGB <cr88192@hotmail.com> wrote:
> a nifty tool I have used in some places is object-origin tracking, where 
> every time the allocator is accessed, it records where it was called 
> from, and will use this information (combined with some data-forensics) 
> to try to make an educated guess as to "who done it" (or, IOW, around 
> where the offending code might be).

I hacked this into the Linux kernel on my last job. The page allocator
would do a backtrace and I took a vector of the addresses (not all of them,
like the last five) and hashed them into a table.

The object of that "who done it" was to track down who (or what) was eating
up the lion's share of pages.

This was viewable in real time in a /proc entry.

[toc] | [prev] | [next] | [standalone]

#14249

In comp.lang.c BGB <cr88192@hotmail.com> wrote:
> On 2/17/2012 3:51 PM, Joshua Maurice wrote:
<snip>
> >> For raw pointers it is more difficult, you can use something like
> >> ElectricFence or Valgrind on Linux, but it makes your program to run
> >> many times slower and consume lots of more memory so this is just for
> >> debugging.
> >
>
> I was mildly aware of ElectricFence and Valgrind, and was also thinking
> mostly of malloc'ed raw pointers and C-style arrays (when passed as
> pointers). the idea would be if something like Valgrind were directly
> integrated into the compiler/runtime as a debug option. this is mostly
> what I was writing about.

Integrating Valgrind into the runtime isn't really practical. Valgrind is a
virtual machine. It'd be like integrating QEMU into the runtime.

But as for Valgrind being too slow and only useful for debugging, I just
can't agree. Valgrind is excellent for _detecting_ bugs, not just diagnosing
them. If more people ran their regression tests on top of Valgrind (i.e.
`valgrind foo` instead of `foo') then the world of software would be a far
better place. I mean no offense, but frankly it boggles my mind how people
excuse themselves from running Valgrind, instead spending countless hours
tinkering with virtual memory paging tricks. (To be fair, I see it all the
time at work, too.)

Granted, when compared to C running C++ apps with Valgrind is slower still.
But that's because C++ apps are less judicious about object manipulation and
tend to use dynamic memory for every teeny-tiny allocation. All the more
reason to spot bugs earlier rather than later, as dynamic memory bugs can be
more maddening than simple stack overflows.

<snip>
> > MSVC also has some debug options that try to catch writes past the
> > ends of allocated regions as well. IIRC, they overallocate, and put
> > special bit patterns at the start and end on allocation, and when
> > freed they check to see if those bit patterns are intact, raising a
> > fatal error or something if it finds a problem.

> this is partly what my memory allocators do, but may also under certain 
> cases scan the heap, detect corruption, and attempt to diagnose it.

> a nifty tool I have used in some places is object-origin tracking, where 
> every time the allocator is accessed, it records where it was called 
> from, and will use this information (combined with some data-forensics) 
> to try to make an educated guess as to "who done it" (or, IOW, around 
> where the offending code might be).

> although, sadly, there is no good way to implement a HW write barrier 
> for this (sadly, neither Windows nor Linux give enough control over the 
> CPU to really make something like this all that workable, even then it 
> would still likely be page-fault driven slowness).

> I have some stuff for "software write barriers" (mostly needed for other 
> reasons), but not a lot of code uses them (unless forced into it), 
> partly due to the added awkwardness and performance overheads.

> one option that could sort of work for larger arrays would be to put 
> unused pages between memory objects, such that going out of bounds is 
> more likely to trigger a page fault.

Valgrind solves all of these issues by using a virtual machine. Then it's
relatively trivial to interpose CPU instructions to, e.g., introduce write
barriers. You can also instrument your code to communicate with Valgrind
during runtime.

[toc] | [prev] | [next] | [standalone]

#14254

On 2/18/2012 5:43 PM, William Ahern wrote:
> In comp.lang.c BGB<cr88192@hotmail.com>  wrote:
>> On 2/17/2012 3:51 PM, Joshua Maurice wrote:
> <snip>
>>>> For raw pointers it is more difficult, you can use something like
>>>> ElectricFence or Valgrind on Linux, but it makes your program to run
>>>> many times slower and consume lots of more memory so this is just for
>>>> debugging.
>>>
>>
>> I was mildly aware of ElectricFence and Valgrind, and was also thinking
>> mostly of malloc'ed raw pointers and C-style arrays (when passed as
>> pointers). the idea would be if something like Valgrind were directly
>> integrated into the compiler/runtime as a debug option. this is mostly
>> what I was writing about.
>
> Integrating Valgrind into the runtime isn't really practical. Valgrind is a
> virtual machine. It'd be like integrating QEMU into the runtime.
>

not literally integrating Valgrind, but more as a debug option for every 
time an assignment takes place through a raw pointer (serving a similar 
role), the runtime will validate that it was a valid write.

it would be slow, but could be sane as a debug feature.


> But as for Valgrind being too slow and only useful for debugging, I just
> can't agree. Valgrind is excellent for _detecting_ bugs, not just diagnosing
> them. If more people ran their regression tests on top of Valgrind (i.e.
> `valgrind foo` instead of `foo') then the world of software would be a far
> better place. I mean no offense, but frankly it boggles my mind how people
> excuse themselves from running Valgrind, instead spending countless hours
> tinkering with virtual memory paging tricks. (To be fair, I see it all the
> time at work, too.)
>

my main issue with it is that it isn't available for Windows AFAIK.


> Granted, when compared to C running C++ apps with Valgrind is slower still.
> But that's because C++ apps are less judicious about object manipulation and
> tend to use dynamic memory for every teeny-tiny allocation. All the more
> reason to spot bugs earlier rather than later, as dynamic memory bugs can be
> more maddening than simple stack overflows.
>

possibly.

> <snip>
>>> MSVC also has some debug options that try to catch writes past the
>>> ends of allocated regions as well. IIRC, they overallocate, and put
>>> special bit patterns at the start and end on allocation, and when
>>> freed they check to see if those bit patterns are intact, raising a
>>> fatal error or something if it finds a problem.
>
>> this is partly what my memory allocators do, but may also under certain
>> cases scan the heap, detect corruption, and attempt to diagnose it.
>
>> a nifty tool I have used in some places is object-origin tracking, where
>> every time the allocator is accessed, it records where it was called
>> from, and will use this information (combined with some data-forensics)
>> to try to make an educated guess as to "who done it" (or, IOW, around
>> where the offending code might be).
>
>> although, sadly, there is no good way to implement a HW write barrier
>> for this (sadly, neither Windows nor Linux give enough control over the
>> CPU to really make something like this all that workable, even then it
>> would still likely be page-fault driven slowness).
>
>> I have some stuff for "software write barriers" (mostly needed for other
>> reasons), but not a lot of code uses them (unless forced into it),
>> partly due to the added awkwardness and performance overheads.
>
>> one option that could sort of work for larger arrays would be to put
>> unused pages between memory objects, such that going out of bounds is
>> more likely to trigger a page fault.
>
> Valgrind solves all of these issues by using a virtual machine. Then it's
> relatively trivial to interpose CPU instructions to, e.g., introduce write
> barriers. You can also instrument your code to communicate with Valgrind
> during runtime.

yeah.

a downside here though is being primarily a Windows developer.

[toc] | [prev] | [next] | [standalone]

#14255

BGB <cr88192@hotmail.com> writes:
>> Valgrind solves all of these issues by using a virtual machine. Then it's
>> relatively trivial to interpose CPU instructions to, e.g., introduce write
>> barriers. You can also instrument your code to communicate with Valgrind
>> during runtime.
>
> yeah.
>
> a downside here though is being primarily a Windows developer.

Would the work to port valgrind to windows be more than the work to
add some vaguely similar functionality to the compiler though?

-miles

-- 
People who are more than casually interested in computers should have at
least some idea of what the underlying hardware is like.  Otherwise the
programs they write will be pretty weird.  -- Donald Knuth

[toc] | [prev] | [next] | [standalone]

#14259

In comp.lang.c Miles Bader <miles@gnu.org> wrote:
> BGB <cr88192@hotmail.com> writes:
> >> Valgrind solves all of these issues by using a virtual machine. Then it's
> >> relatively trivial to interpose CPU instructions to, e.g., introduce write
> >> barriers. You can also instrument your code to communicate with Valgrind
> >> during runtime.
> >
> > yeah.
> >
> > a downside here though is being primarily a Windows developer.

> Would the work to port valgrind to windows be more than the work to
> add some vaguely similar functionality to the compiler though?

FWIW, 

	"Windows is not under consideration here because porting to it would
	require so many changes it would almost be a separate project.
	(However, Valgrind + Wine can be made to work with some effort.)
	Also, non-open-source OSes are difficult to deal with; being able to
	see the OS and associated (libc) source code makes things much
	easier. However, Valgrind is quite usable in conjunction with Wine,
	which means that it is possible to run Windows programs under
	Valgrind with some effort."

	-- http://valgrind.org/info/platforms.html

[toc] | [prev] | [next] | [standalone]

#14288

"Miles Bader" <miles@gnu.org> ha scritto nel messaggio
news:874nun1nzs.fsf@catnip.gol.com...
> -- 
> People who are more than casually interested in computers should have at
> least some idea of what the underlying hardware is like.  Otherwise the
> programs they write will be pretty weird.  -- Donald Knuth

i think in the future cpus would be very similar
all with more core and different kind of registers
for dimention 8bit 16bit 32bit 64bit 128bit etc

[toc] | [prev] | [next] | [standalone]

#14258

In comp.lang.c BGB <cr88192@hotmail.com> wrote:
> On 2/18/2012 5:43 PM, William Ahern wrote:
> > In comp.lang.c BGB<cr88192@hotmail.com>  wrote:
> >> On 2/17/2012 3:51 PM, Joshua Maurice wrote:
> > <snip>
> >>>> For raw pointers it is more difficult, you can use something like
> >>>> ElectricFence or Valgrind on Linux, but it makes your program to run
> >>>> many times slower and consume lots of more memory so this is just for
> >>>> debugging.
> >>>
> >>
> >> I was mildly aware of ElectricFence and Valgrind, and was also thinking
> >> mostly of malloc'ed raw pointers and C-style arrays (when passed as
> >> pointers). the idea would be if something like Valgrind were directly
> >> integrated into the compiler/runtime as a debug option. this is mostly
> >> what I was writing about.
> >
> > Integrating Valgrind into the runtime isn't really practical. Valgrind
> > is a virtual machine. It'd be like integrating QEMU into the runtime.
> >

> not literally integrating Valgrind, but more as a debug option for every 
> time an assignment takes place through a raw pointer (serving a similar 
> role), the runtime will validate that it was a valid write.

> it would be slow, but could be sane as a debug feature.

Valgrind maintains a bitmap of every byte of allocated memory. It can verify
every read and write operation at the [virtual] hardware level, including
whether a byte has been read from before being written to.

Although there's a tradeoff when filtering out the tremendous amount of
noise--usually involving stack or syscall operations--wherein occassionaly
some bugs are let through without a warning. This is almost never an issue
when dealing with dynamic memory from malloc() or new.

But in order to do all of these things it needs to watch everything from the
very beginning. You can't attach Valgrind to a running app; rather, you use
it to execute the application from the start. Instead of executing
`foo -bar', you execute `valgrind foo -bar'. In that way it's trivial to
integrate Valgrind into an IDE.

> > But as for Valgrind being too slow and only useful for debugging, I just
> > can't agree. Valgrind is excellent for _detecting_ bugs, not just
> > diagnosing them. If more people ran their regression tests on top of
> > Valgrind (i.e. `valgrind foo` instead of `foo') then the world of
> > software would be a far better place. I mean no offense, but frankly it
> > boggles my mind how people excuse themselves from running Valgrind,
> > instead spending countless hours tinkering with virtual memory paging
> > tricks. (To be fair, I see it all the time at work, too.)
> >

> my main issue with it is that it isn't available for Windows AFAIK.

That is, admittedly, a pretty decent excuse. Although, I primarily target
OpenBSD, which Valgrind has yet to support. This is one of those contexts
where code portability is priceless, apropos comp.lang.c and comp.lang.c++!
The real barrier, IME, isn't the code per se, but the build environment. But
writing portable code helps there as well, with less reliance on compiler
and build environment magic.

[toc] | [prev] | [next] | [standalone]

#14262

On 2/18/2012 10:06 PM, William Ahern wrote:
> In comp.lang.c BGB<cr88192@hotmail.com>  wrote:
>> On 2/18/2012 5:43 PM, William Ahern wrote:
>>> In comp.lang.c BGB<cr88192@hotmail.com>   wrote:
>>>> On 2/17/2012 3:51 PM, Joshua Maurice wrote:
>>> <snip>
>>>>>> For raw pointers it is more difficult, you can use something like
>>>>>> ElectricFence or Valgrind on Linux, but it makes your program to run
>>>>>> many times slower and consume lots of more memory so this is just for
>>>>>> debugging.
>>>>>
>>>>
>>>> I was mildly aware of ElectricFence and Valgrind, and was also thinking
>>>> mostly of malloc'ed raw pointers and C-style arrays (when passed as
>>>> pointers). the idea would be if something like Valgrind were directly
>>>> integrated into the compiler/runtime as a debug option. this is mostly
>>>> what I was writing about.
>>>
>>> Integrating Valgrind into the runtime isn't really practical. Valgrind
>>> is a virtual machine. It'd be like integrating QEMU into the runtime.
>>>
>
>> not literally integrating Valgrind, but more as a debug option for every
>> time an assignment takes place through a raw pointer (serving a similar
>> role), the runtime will validate that it was a valid write.
>
>> it would be slow, but could be sane as a debug feature.
>
> Valgrind maintains a bitmap of every byte of allocated memory. It can verify
> every read and write operation at the [virtual] hardware level, including
> whether a byte has been read from before being written to.
>
> Although there's a tradeoff when filtering out the tremendous amount of
> noise--usually involving stack or syscall operations--wherein occassionaly
> some bugs are let through without a warning. This is almost never an issue
> when dealing with dynamic memory from malloc() or new.
>
> But in order to do all of these things it needs to watch everything from the
> very beginning. You can't attach Valgrind to a running app; rather, you use
> it to execute the application from the start. Instead of executing
> `foo -bar', you execute `valgrind foo -bar'. In that way it's trivial to
> integrate Valgrind into an IDE.
>

I would have assumed coarser checking, maybe in 16-byte units (or, at 
least that would be convenient for me, as my memory manager allocates 
memory in terms of 16-byte cells). actually, for smaller objects, the MM 
allocates via a bitmap as well, but uses free-list allocation for larger 
objects (once one gets into the kB range).

actually, the bulk of memory allocation in my apps tends to be objects 
under 1kB (and many tiny objects exists).


>>> But as for Valgrind being too slow and only useful for debugging, I just
>>> can't agree. Valgrind is excellent for _detecting_ bugs, not just
>>> diagnosing them. If more people ran their regression tests on top of
>>> Valgrind (i.e. `valgrind foo` instead of `foo') then the world of
>>> software would be a far better place. I mean no offense, but frankly it
>>> boggles my mind how people excuse themselves from running Valgrind,
>>> instead spending countless hours tinkering with virtual memory paging
>>> tricks. (To be fair, I see it all the time at work, too.)
>>>
>
>> my main issue with it is that it isn't available for Windows AFAIK.
>
> That is, admittedly, a pretty decent excuse. Although, I primarily target
> OpenBSD, which Valgrind has yet to support. This is one of those contexts
> where code portability is priceless, apropos comp.lang.c and comp.lang.c++!
> The real barrier, IME, isn't the code per se, but the build environment. But
> writing portable code helps there as well, with less reliance on compiler
> and build environment magic.
>

well, my stuff does build on Linux, but given it uses OpenGL, only 
performs at usable speeds if the computer is actually booted from Linux 
(rather than run from VMware).

[toc] | [prev] | [next] | [standalone]

#14282

Hi BGB

BGB wrote:

> > Which is why they are exceedingly rare. Nearly all allocation problems are due
> > to the program storing outside array bounds.
> > 
> 
> which are annoyingly difficult to track down sometimes...
> 
> 
> it would be nice if compilers would offer an option (as a debugging feature) to 
> optionally put in bounds checking for many operations (it is also possible to do 
> this without adding fat pointers or fundamentally changing how the language 
> works, although it would still introduce a run-time cost).

Such a tool have been part of Borland/CodeGear/Embarcadero compilers for years
it is a debug option called CodeGuard, it also checks if You remember to delete 
what You have created with new. 
Just for debugging though, it is to slow for release code.

Best regards
Asger-P

[toc] | [prev] | [next] | [standalone]

#14285

On 2/19/2012 8:54 AM, Asger Joergensen wrote:
> Hi BGB
>
> BGB wrote:
>
>>> Which is why they are exceedingly rare. Nearly all allocation problems are due
>>> to the program storing outside array bounds.
>>>
>>
>> which are annoyingly difficult to track down sometimes...
>>
>>
>> it would be nice if compilers would offer an option (as a debugging feature) to
>> optionally put in bounds checking for many operations (it is also possible to do
>> this without adding fat pointers or fundamentally changing how the language
>> works, although it would still introduce a run-time cost).
>
> Such a tool have been part of Borland/CodeGear/Embarcadero compilers for years
> it is a debug option called CodeGuard, it also checks if You remember to delete
> what You have created with new.
> Just for debugging though, it is to slow for release code.
>

yep.

[toc] | [prev] | [next] | [standalone]

#14123

On 2/16/2012 4:21 PM, Devil with the China Blue Dress wrote:
> In article<jhjr14$rjk$1@reader1.panix.com>, jgk@panix.com (Joe keane) wrote:
>
>> Bugs of memory allocation will make you mad.
>>
>> Bugs *in* memory allocation will put you in the cuckoo people place.
>
> Which is why they are exceedingly rare. Nearly all allocation problems are due
> to the program storing outside array bounds.

     The only allocator bug I have personally encountered was with
a malloc() implementation that never, never returned NULL.  When it
ought to have returned NULL, it crashed the program instead ...

-- 
Eric Sosman
esosman@ieee-dot-org.invalid

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.c++

csiph-web