Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #66967 > unrolled thread

Re: I want to ask a question...

Back to article view | Back to comp.lang.c

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: I want to ask a question... Gareth Owen <gwowen@gmail.com> - 2015-08-07 20:00 +0100
    Re: I want to ask a question... glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-08-07 19:04 +0000
    Re: I want to ask a question... Keith Thompson <kst-u@mib.org> - 2015-08-07 15:30 -0700
      Re: I want to ask a question... glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-08-07 22:52 +0000
        Re: I want to ask a question... Keith Thompson <kst-u@mib.org> - 2015-08-07 16:46 -0700
          Re: I want to ask a question... glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-08-07 23:29 +0000
            Re: I want to ask a question... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-08-08 01:37 +0100
            Re: I want to ask a question... Keith Thompson <kst-u@mib.org> - 2015-08-07 18:55 -0700
              Re: I want to ask a question... glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-08-08 05:51 +0000
                Re: I want to ask a question... Robert Wessel <robertwessel2@yahoo.com> - 2015-08-08 01:44 -0500
      Re: I want to ask a question... Ken Brody <kenbrody@spamcop.net> - 2015-08-10 12:46 -0400
      Re: I want to ask a question... supercat@casperkitty.com - 2015-08-19 16:43 -0700
        Re: I want to ask a question... Keith Thompson <kst-u@mib.org> - 2015-08-19 18:09 -0700
          Re: I want to ask a question... glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-08-20 01:00 +0000
            Re: I want to ask a question... Keith Thompson <kst-u@mib.org> - 2015-08-19 20:25 -0700
              Re: I want to ask a question... supercat@casperkitty.com - 2015-08-20 08:15 -0700
                Re: I want to ask a question... Keith Thompson <kst-u@mib.org> - 2015-08-20 11:38 -0700
                  Re: I want to ask a question... glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-08-20 18:47 +0000
                  Re: I want to ask a question... supercat@casperkitty.com - 2015-08-20 14:01 -0700
              Re: I want to ask a question... glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-08-20 16:43 +0000
                Re: I want to ask a question... James Kuyper <jameskuyper@verizon.net> - 2015-08-20 13:02 -0400
                  Re: I want to ask a question... Tonton Th <tTh@nowhere.invalid> - 2015-08-20 17:40 +0000
                    Re: I want to ask a question... James Kuyper <jameskuyper@verizon.net> - 2015-08-20 15:27 -0400
                      Re: I want to ask a question... David Brown <david.brown@hesbynett.no> - 2015-08-20 22:58 +0200
                        Re: I want to ask a question... James Kuyper <jameskuyper@verizon.net> - 2015-08-20 23:24 -0400
                          Re: I want to ask a question... glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-08-21 06:19 +0000
                          Re: I want to ask a question... Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2015-08-21 09:20 -0600
                          Re: I want to ask a question... supercat@casperkitty.com - 2015-08-22 12:25 -0700
                            Re: I want to ask a question... Robert Wessel <robertwessel2@yahoo.com> - 2015-08-23 01:33 -0500
                    Re: I want to ask a question... glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-08-20 19:28 +0000
                Re: I want to ask a question... Keith Thompson <kst-u@mib.org> - 2015-08-20 12:11 -0700
            Re: I want to ask a question... David Brown <david.brown@hesbynett.no> - 2015-08-20 10:02 +0200
          Re: I want to ask a question... raltbos@xs4all.nl (Richard Bos) - 2015-08-22 13:42 +0000
            Re: I want to ask a question... supercat@casperkitty.com - 2015-08-22 12:23 -0700

Page 2 of 2 — ← Prev page 1 [2]

#67890

On 08/20/2015 12:43 PM, glen herrmannsfeldt wrote:
...
> For those using C as a replecement for assembly,

Which is an idiotic thing to do. If you want to specify precisely the
instructions to be generated, use a language whose defined behavior is
to do precisely that - C isn't such a language, and was never intended
to be.

> ... they often know
> exactly what the generated code will look like. ...

More accurately, they "assume they know".

> I am completely against programs that dereference pointers that
> aren't pointing to the right place, but not so much against ones
> that copy or compare pointers.

Why? It's undefined behavior either way, if the the pointer has an
indeterminate value, and the standard was written that way because there
are real systems on which comparison of a pointer with and indeterminate
value could have nasty consequences.
Copying with memcpy() should be safe even when the value of the pointer
is indeterminate, but it's hard to come up with a justification for
doing so, since it's just as much undefined behavior to do anything
meaningful with the copy as it was with the original.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#67896

On 2015-08-20, James Kuyper <jameskuyper@verizon.net> wrote:

> Why? It's undefined behavior either way, if the the pointer has an
> indeterminate value, and the standard was written that way because there
> are real systems on which comparison of a pointer with and indeterminate
> value could have nasty consequences.
  
   Can you give us a real (and may be historic) exemple of
   some systems where this is the case ?


-- 
<<<     http://la.buvette.org/photos/myrys/20ans/discours-corporate.ogg

[toc] | [prev] | [next] | [standalone]

#67904

On 08/20/2015 01:40 PM, Tonton Th wrote:
> On 2015-08-20, James Kuyper <jameskuyper@verizon.net> wrote:
> 
>> Why? It's undefined behavior either way, if the the pointer has an
>> indeterminate value, and the standard was written that way because there
>> are real systems on which comparison of a pointer with and indeterminate
>> value could have nasty consequences.
>   
>    Can you give us a real (and may be historic) exemple of
>    some systems where this is the case ?

I remember systems with unusual features mainly for the purpose of
remembering that those features actually exist, so I tend not to
remember the corresponding system names, so I can't name it.
However, I can describe the features the system had that made this a
problem, and maybe someone will recognize the description. It had a
virtual memory system, with ranges of address space being associated
with an arbitrary blocks of physical memory. It kept track of which
range of addresses was associated with each block of memory currently in
use, and trapped upon any attempt to use an address (even just in a
comparison for equality) not currently allocated to any physical block
of memory. Calling free() could result in a block being removed from the
list of blocks currently in use.
It's that last feature that is, apparently, the least common - I've been
told that most implementations of the malloc() family don't actually
release memory back to the system until the program exits, which seems
like rather selfish behavior to me.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#67909

On 20/08/15 21:27, James Kuyper wrote:
> On 08/20/2015 01:40 PM, Tonton Th wrote:
>> On 2015-08-20, James Kuyper <jameskuyper@verizon.net> wrote:
>>
>>> Why? It's undefined behavior either way, if the the pointer has an
>>> indeterminate value, and the standard was written that way because there
>>> are real systems on which comparison of a pointer with and indeterminate
>>> value could have nasty consequences.
>>
>>     Can you give us a real (and may be historic) exemple of
>>     some systems where this is the case ?
>
> I remember systems with unusual features mainly for the purpose of
> remembering that those features actually exist, so I tend not to
> remember the corresponding system names, so I can't name it.
> However, I can describe the features the system had that made this a
> problem, and maybe someone will recognize the description. It had a
> virtual memory system, with ranges of address space being associated
> with an arbitrary blocks of physical memory. It kept track of which
> range of addresses was associated with each block of memory currently in
> use, and trapped upon any attempt to use an address (even just in a
> comparison for equality) not currently allocated to any physical block
> of memory. Calling free() could result in a block being removed from the
> list of blocks currently in use.
> It's that last feature that is, apparently, the least common - I've been
> told that most implementations of the malloc() family don't actually
> release memory back to the system until the program exits, which seems
> like rather selfish behavior to me.
>

I believe it is common for malloc implementations to ask the OS for 
large chunks of memory, then dish out parts of that as needed by the 
program.  This way they only bother the OS on occasion, and the overhead 
for malloc is much smaller (there are no context switches or system 
calls for most allocations).  Memory can only be returned to the OS if a 
whole chunk has been freed - and if the program has a mix of malloc's 
and free's, that is rarely going to happen.  It needs a program that has 
allocated a substantial amount of memory, freed it, and then continues 
running for a significant time without re-using that memory space. 
Maybe it is simply not worth the overhead supporting this.

[toc] | [prev] | [next] | [standalone]

#67921

On 08/20/2015 04:58 PM, David Brown wrote:
...
> I believe it is common for malloc implementations to ask the OS for 
> large chunks of memory, then dish out parts of that as needed by the 
> program.  This way they only bother the OS on occasion, and the overhead 
> for malloc is much smaller (there are no context switches or system 
> calls for most allocations).  Memory can only be returned to the OS if a 
> whole chunk has been freed - ...

I agree with all of the above.

> ... and if the program has a mix of malloc's 
> and free's, that is rarely going to happen.  It needs a program that has 
> allocated a substantial amount of memory, freed it, and then continues 
> running for a significant time without re-using that memory space.

That doesn't strike me as a particular odd thing for a program to do.
I've seen programs that have a memory-intensive set-up phase, after
which they spend a lot of time waiting (sometimes for days or weeks at a
time) for events to happen and reacting to them, without needing
anywhere near as much memory. If I were using such a program, I'd be
very upset that it was monopolizing it's entire initial memory
requirement for the entire life of the program. Of course, a lot of
modern software hogs a lot of memory, and possibly this is one of the
reasons why.

That pattern is not typical of the programs I write: they aren't event
driven, and typically have a set-up phase that uses far less memory than
the main processing phase, not more. But I'm very aware that other
patterns exist.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#67922

James Kuyper <jameskuyper@verizon.net> wrote:
> On 08/20/2015 04:58 PM, David Brown wrote:
> ...
>> I believe it is common for malloc implementations to ask the OS for 
>> large chunks of memory, then dish out parts of that as needed by the 
>> program.  This way they only bother the OS on occasion, and the overhead 
>> for malloc is much smaller (there are no context switches or system 
>> calls for most allocations).  Memory can only be returned to the OS if a 
>> whole chunk has been freed - ...

> I agree with all of the above.
 
(snip)

> That doesn't strike me as a particular odd thing for a program to do.
> I've seen programs that have a memory-intensive set-up phase, after
> which they spend a lot of time waiting (sometimes for days or weeks at a
> time) for events to happen and reacting to them, without needing
> anywhere near as much memory. If I were using such a program, I'd be
> very upset that it was monopolizing it's entire initial memory
> requirement for the entire life of the program. Of course, a lot of
> modern software hogs a lot of memory, and possibly this is one of the
> reasons why.

For most systems, that is virtual memory, part of each tasks own
virtual address space. 

There have been systems with a single virtual address space for
all tasks, but that is rare now. 

Other than that, different systems allocate real memory to tasks
in different ways, some of which may be more or less efficient in
different ways. 

How each program uses its own virtual address space can make a
difference. Fragmentation in virtual space can result in more
paging, and otherwise waste resources. 

-- glen

[toc] | [prev] | [next] | [standalone]

#67948

James Kuyper <jameskuyper@verizon.net> writes:

> On 08/20/2015 04:58 PM, David Brown wrote:
> ...
>> I believe it is common for malloc implementations to ask the OS for 
>> large chunks of memory, then dish out parts of that as needed by the 
>> program.  This way they only bother the OS on occasion, and the overhead 
>> for malloc is much smaller (there are no context switches or system 
>> calls for most allocations).  Memory can only be returned to the OS if a 
>> whole chunk has been freed - ...
>
> I agree with all of the above.
>
>> ... and if the program has a mix of malloc's 
>> and free's, that is rarely going to happen.  It needs a program that has 
>> allocated a substantial amount of memory, freed it, and then continues 
>> running for a significant time without re-using that memory space.
>
> That doesn't strike me as a particular odd thing for a program to do.
> I've seen programs that have a memory-intensive set-up phase, after
> which they spend a lot of time waiting (sometimes for days or weeks at a
> time) for events to happen and reacting to them, without needing
> anywhere near as much memory. If I were using such a program, I'd be
> very upset that it was monopolizing it's entire initial memory
> requirement for the entire life of the program.

The stuff that was used at startup and never again gets paged out to the
swap space.  No problem.

[toc] | [prev] | [next] | [standalone]

#68035

On Thursday, August 20, 2015 at 10:24:54 PM UTC-5, James Kuyper wrote:
> That doesn't strike me as a particular odd thing for a program to do.
> I've seen programs that have a memory-intensive set-up phase, after
> which they spend a lot of time waiting (sometimes for days or weeks at a
> time) for events to happen and reacting to them, without needing
> anywhere near as much memory. If I were using such a program, I'd be
> very upset that it was monopolizing it's entire initial memory
> requirement for the entire life of the program. Of course, a lot of
> modern software hogs a lot of memory, and possibly this is one of the
> reasons why.

Generally if software is going to have a transient need for a lot of memory,
it should request a block of memory directly from the OS, use that block to
satisfy the transient needs, and then free that block of memory back to the
OS when it's done with it.  The malloc/free mechanism built into C was designed
to be simple and sufficient for many purposes, but really efficient memory
management often requires consolidating allocations with similar lifetimes, and
malloc/free have no way to do that.

[toc] | [prev] | [next] | [standalone]

#68094

On Sat, 22 Aug 2015 12:25:29 -0700 (PDT), supercat@casperkitty.com
wrote:

>On Thursday, August 20, 2015 at 10:24:54 PM UTC-5, James Kuyper wrote:
>> That doesn't strike me as a particular odd thing for a program to do.
>> I've seen programs that have a memory-intensive set-up phase, after
>> which they spend a lot of time waiting (sometimes for days or weeks at a
>> time) for events to happen and reacting to them, without needing
>> anywhere near as much memory. If I were using such a program, I'd be
>> very upset that it was monopolizing it's entire initial memory
>> requirement for the entire life of the program. Of course, a lot of
>> modern software hogs a lot of memory, and possibly this is one of the
>> reasons why.
>
>Generally if software is going to have a transient need for a lot of memory,
>it should request a block of memory directly from the OS, use that block to
>satisfy the transient needs, and then free that block of memory back to the
>OS when it's done with it.  The malloc/free mechanism built into C was designed
>to be simple and sufficient for many purposes, but really efficient memory
>management often requires consolidating allocations with similar lifetimes, and
>malloc/free have no way to do that.


And new in C++ has the same issues.  You can usually supply custom
allocators in C++, but it's often a far from trivial undertaking.
Garbage collected languages have a real advantage here (for example,
many allocators for Java will track a ratio of used-to-allocated
memory after GC's and if a limit is exceeded, release some of the
storage back to the OS).

[toc] | [prev] | [next] | [standalone]

#67905

Tonton Th <tTh@nowhere.invalid> wrote:
> On 2015-08-20, James Kuyper <jameskuyper@verizon.net> wrote:
 
>> Why? It's undefined behavior either way, if the the pointer has an
>> indeterminate value, and the standard was written that way because there
>> are real systems on which comparison of a pointer with and indeterminate
>> value could have nasty consequences.
  
>   Can you give us a real (and may be historic) exemple of
>   some systems where this is the case ?

It depends on what is nasty. In large mode 80286, pointer arithmetic
is done on the offset, but not the selector. That is a different case
than here, but is related to the standard allowing comparison one
element beyond the end of an array being pointed to.

   for(p=a; p<a+n; p++) putchar(*p);

For large mode 80286, a can't be 65536 bytes long, as the pointer
will wrap, and the comparison fail. Maybe about as surprising:

   unsigned i;
   for(i=9;i>=0;i--) printf("%u\n",i);

I believe compilers will warn about this, maybe not the previous one.

-- glen

[toc] | [prev] | [next] | [standalone]

#67897

glen herrmannsfeldt <gah@ugcs.caltech.edu> writes:
[...]
> I am completely against programs that dereference pointers that
> aren't pointing to the right place, but not so much against ones
> that copy or compare pointers.

I hope you can find a language that agrees with your preferences.
It isn't C.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#67874

On 20/08/15 03:00, glen herrmannsfeldt wrote:
> Keith Thompson <kst-u@mib.org> wrote:
>> supercat@casperkitty.com writes:
> 
> (snip)
>>> If a pointer isn't null, doesn't identify a valid object, and isn't a
>>> "one-past" pointer of a valid object, even something as seemingly-benign as
> 
>>>     free(ptr);
>>>     if (condition1) {foo(); ptr=NULL;}
>>>     if (condition2) {bar(); ptr=NULL;}
>>>     if (ptr != NULL) printf("Forgot to clear out pointer!");
> 
>>> may cause some compilers to infer that if condition1 is false, bar()
>>> should be called unconditionally since the Standard would not require
>>> that the compiler evaluate condition2 in such case.  Unfortunately,
>>> some compilers writers seem to think that such "optimizations" are a
>>> good thing.
>  
>> The code has undefined behavior if condition1 and condition2 are both
>> false.
> 
> I completely agree that it is official UB.
> 
>> Since the code is incorrect, it shouldn't matter much how it behaves.
>> But it would be nice if the compiler printed a warning.
> 
> I suppose a warning is fine, but on most processors it isn't
> all that hard to do what the program says. Doing the optimization
> and then not warning about it is mean. Legal, but mean.
> 
> How about a special option that enables this kind of optimization,
> so it can be turned off when one doesn't want it.
> 

While such warnings or optimisation options sound nice, it quickly
becomes extremely difficult to handle correctly.  You could make it work
on simple test examples, but complex code - where you /really/ need such
features, as the UB may be hard to spot - become intractable.

Compilers usually do not "take advantage" of UB in code generation
unless there is a lot to be gained - if it is just as easy to generate
code that works the way the programmer seems to intend, then compilers
will (in my experience - though hopefully the UB is not in /my/ code!)
generate the results the programmer expected.  But it is, I think,
impossible to properly define and delineate how the compiler should
handle such situations.

And it's always great when a compiler can warn of UB issues in the code
- but again, there is no way for the compiler to detect all such cases
as this.

[toc] | [prev] | [next] | [standalone]

#68008

Keith Thompson <kst-u@mib.org> wrote:

> supercat@casperkitty.com writes:

> > If a pointer isn't null, doesn't identify a valid object, and isn't a
> > "one-past" pointer of a valid object, even something as seemingly-benign as
> >
> >     free(ptr);
> >     if (condition1) {foo(); ptr=NULL;}
> >     if (condition2) {bar(); ptr=NULL;}
> >     if (ptr != NULL) printf("Forgot to clear out pointer!");
> >
> > may cause some compilers to infer that if condition1 is false, bar()
> > should be called unconditionally since the Standard would not require
> > that the compiler evaluate condition2 in such case.  Unfortunately,
> > some compilers writers seem to think that such "optimizations" are a
> > good thing.
> 
> The code has undefined behavior if condition1 and condition2 are both
> false.
> 
> Since the code is incorrect, it shouldn't matter much how it behaves.
> But it would be nice if the compiler printed a warning.

Careful, though! It's _only_ incorrect, and the "optimisation" may
_only_ be made, if the compiler can prove that both conditions _must_ be
false.
It's allowable for there to be code that does something weird at the
fourth line if ptr is still indeterminate, but it's not allowable for
this weirdness to happen before this comparison is made, just on the
off-chance. Only if it's provable that it _will_ be indeterminate may
the UB happen earlier, not if it _may_ be.
After all, it's quite possible (even if inadvisable) to write code like
this where the programmer knows that either condition1 or condition2 (or
possibly both) always holds, even if the compiler cannot prove this.

Richard

[toc] | [prev] | [next] | [standalone]

#68034

On Saturday, August 22, 2015 at 8:43:00 AM UTC-5, Richard Bos wrote:
> > supercat writes:
> > >     free(ptr);
> > >     if (condition1) {foo(); ptr=NULL;}
> > >     if (condition2) {bar(); ptr=NULL;}
> > >     if (ptr != NULL) printf("Forgot to clear out pointer!");

> > The code has undefined behavior if condition1 and condition2 are both
> > false.

> Careful, though! It's _only_ incorrect, and the "optimisation" may
> _only_ be made, if the compiler can prove that both conditions _must_ be
> false.

If the first condition is false, the program can only proceed without
invoking Undefined Behavior if the second condition is true.  Hyper-modern
philosophy would thus dictate that the code should be optimized to:

    free(ptr);
    if (!condition1) goto EXEC2;
    foo();
    if (condition2)
    {
    EXEC2:
      bar();
    }
    ptr = null; // May be omitted if ptr is not read before next write

The optimization is legitimate even if condition1 is false and condition2 is
true, since although it will skip the test on condition2, it will only execute
code which was required to execute anyway.

> It's allowable for there to be code that does something weird at the
> fourth line if ptr is still indeterminate, but it's not allowable for
> this weirdness to happen before this comparison is made, just on the
> off-chance. Only if it's provable that it _will_ be indeterminate may
> the UB happen earlier, not if it _may_ be.

If condition1 is false, the fact that the test on condition2 is skipped will
only be observable if condition2 were also false, invoking Undefined Behavior.
Under the as-if rule, a compiler would thus be allowed to omit the test on
condition2.

Note that I do not think that compilers should do this; indeed, I would suggest
that there should be a way for a program to assert "Any compiler which cannot
guarantee that an equality-comparison between a pointer to a freed object and
"null" will have no effect other than to report the pointer unequal, must consider this program ill-formed", in which case the above code would not
invoke Undefined Behavior (since the only compilers that would be allowed to
accept it would be those which defined a behavior).

One thing I find particularly vexatious with this form of UB is that tests
like the above are often considered good defensive programming practice, but
hyper-modern optimizations may not only result in the assertion-checking code
getting omitted (meaning execution might proceed past the point where the
errant state should have been stopped and trapped if I'd remembered to include
an exit(2) after the printf), but the existence of the assertion may make
the effects of the errant state far more difficult to diagnose than they would
have been in its absence.

> After all, it's quite possible (even if inadvisable) to write code like
> this where the programmer knows that either condition1 or condition2 (or
> possibly both) always holds, even if the compiler cannot prove this.

The indicated optimization would not affect correctness in an defined
scenario. What would be affected would be the consequences of errant state;
it may be that without the assertion logic above, the effect of errant state
would have been to read (but never write) memory which was unallocated, an
act which on most platforms will either kill the program or yield arbitrary
data.  If the read were to trap, that would have indicated a problem, and
yielding arbitrary data might have indicated a problem or might have been
harmless.  The effect of executing bar() when condition2 is false, however,
might not be so benign.  Further, a programmer who is told that bar() was
erroneously executed, would be apt to be far more focused on trying to figure
out why condition2 was erroneously true, rather than on why condition1 may
have been erroneously false [the fact that `bar` executed and the `printf` did
not would surely point much more strongly to the former conclusion!].

I wonder to what extent these sorts of compiler optimizations are used in
production code by people who are informed of how they can transform code.
Unless it would be possible following an unexpected failure in a production
system to feed a debug build the exact same inputs that led to the failure,
I can't see any way by which someone could possibly hope to diagnose a problem
like the above, especially if the above pattern were hidden in a larger piece
of code.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c

csiph-web