Re: Optimization techniques and undefined behavior

From	David Brown <david.brown@hesbynett.no>
Newsgroups	comp.compilers
Subject	Re: Optimization techniques and undefined behavior
Date	2019-05-06 08:14 +0200
Organization	A noiseless patient Spider
Message-ID	<19-05-033@comp.compilers> (permalink)
References	(6 earlier) <19-04-047@comp.compilers> <19-05-004@comp.compilers> <19-05-006@comp.compilers> <19-05-016@comp.compilers> <19-05-017@comp.compilers>

Show all headers | View raw

On 03/05/2019 11:52, Martin Ward wrote:
> On 03/05/19 00:48, Bart wrote:
>> And I think that if a program can
>> go seriously wrong through unchecked input, then that's a failure in
>> proper validation. It's rather sloppy to rely on a runtime check put
>> their by a compiler.
>
> The car analogy for C is that C is a car with no seatbelts, crumple
> zones, roll bars, airbags etc.  The car manual explicitly states that
> nudging the kerb with any tyre is "undefined behaviour" and could
> cause the car to explode in a fireball, killing all the passengers.

That is not /quite/ right.  The C standards are like laws saying that a
car has to have an engine and a number of wheels (but that is up to the
manufacturer to decide how many, as long as they tell you).  The
regulations don't require seatbelts, but you can always make your own -
and a manufacturer is free to add any extras they like as long as they
don't hinder the basic function of the car.  The manufacturer can also
change details of how the car works depending on who is driving it,
trimming the engine or adding ABS brakes according to preference.

>
> On 2019-05-01, David Brown <david.brown@hesbynett.no> wrote:
>> Detecting signed overflow at run-time can be a significant cost.
>
> Firstly: the cost is not as high as the cost of security breaches due
> to buffer overflows.

Apples and oranges.

The cost of adding run-time costs can often be significant, and in
extreme cases it simply means the code can't be written in that language
- C is an appropriate choice of programming language precisely when you
want maximal efficiency.  And this cost applies even when the programmer
knows fine that the operations don't overflow.

The consequences of bugs can be severe - of that there is no doubt.  But
run-time overflow checking does not stop security bugs - it is unlikely
to make a significant dent in them.

It is clearly absurd to take an example of a security bug (like the
libpng one), blame it all on a minor part (the fault was failing to
check unknown data, nothing to do with overflows), and then take that as
"proof" that run-time overflow checking will solve security problems.

Security problems are caused by /bugs/.  That can be bugs in the
specification of the program, bugs in the design, bugs in the
management, just as much as bugs in the coding.  Bugs in the coding can
come from all sorts of sources.  Overflows (whether they are arithmetic
overflows, or buffer overflows) are just one of many types of potential
bugs.

Now, as I have said before, it makes a lot of sense to enable run-time
checking while testing and debugging, to find and eliminate as many bugs
as possible.  That does not mean you want the checks in after you have
tested the code - if your testing is good, your development process is
good, and your code review process is good, then it is highly unlikely
that there are such overflows left in the code.  Having run-time checks
means bigger and slower code, and lots of code paths that never get
tested - which is a really bad idea.

And often it would simply be better to use a different language, rather
than C.

> Secondly: if many popular languages specified
> suitable handling for signed overflow, buffer overruns and so on, then
> CPUs hardware would be developed which makes these tests efficient:
> because compiled code in these popular languages would run faster on
> such CPUs.

Signed overflow and buffer overflow have little in common, expect
coincidental names.  You need to consider them separately, unless you
are just talking about bugs in general.

The inefficiency of signed overflow detection is not a matter of cpu
instructions.  Quite a few cpus have operations with "trap on overflow"
behaviour, or at least a "trap if overflow flag set" operation.  The
major cost, in many cases, is in how it hinders re-arrangements of
expressions, common expression elimination, simplifications, etc.  A
second case is that it severely limits the use SIMD or vector operations.

Another point is that it rarely matches reality.  What is so special
about 2147483647 that you want to make sure "a + b" does not exceed it?
It is more realistic that the limit is something entirely different.
Maybe you are talking about percentages, and the limit is 100.  When you
have realistic requirements based on the code and the problem, rather
than arbitrary limits based on some implementation type, you can quickly
see that this would be hard to handle in cpu instructions.

For buffer overflows, it can be both easier and harder.  It is possible
to have hardware that attaches a size to a pointer, and does some
checking.  It would be quite simple to check addressing modes of "base
register + index" style, but a lot harder when addresses are
pre-computed in other registers.  You can do it if you ensure that the
logical addresses of data objects are in distinct areas, but that puts a
lot of overhead on the OS and memory allocators, and accesses involve a
great deal more virtual page lookups.  It is used by tools such as
valgrind or address sanitizers - tools you want during development and
testing to find the bugs, but not to have on released code.

There have been many attempts through the ages to add special features
in processors to aid security, run-time error checking, supporting
checked languages (like Java), etc.  Most of them have been failures.
You are not the first person to think of this.

>
>> I was talking about a /dimension/ of 2 billion - that is, a width or
>> height of 2 billion.
>
> If you are reading from an unknown file (eg an image on a web page)
> then it would be foolish to assume that no dimension is bigger that 2
> billion: security breaches due to carefully constructed image files
> have occurred in the past.

You missed my point completely.  When I say that the image file should
not have a dimension of such ridiculous sizes, I mean you should check
the dimensions given by the file, and reject the file as broken if its
dimensions exceed such sizes.  I repeatedly said you must check the
unknown data for sanity.

>  Also, the netpbm library can be used for
> files containing data which is *not* image data: for example, as
> generic utilities for processing huge bit strings.  These bit strings
> might well contain more than 2 billion bits (250 MB of data).
>

It does not matter - the principle is the same.  Check the unknown data
for sanity.  The sizes of the data will not be so big that you will be
overflowing properly chosen types (in C or whatever language you want).
You can do these checks safely and without overflows.

> Back in the early days of Unix there were many utilities for
> processing text files.  It was discovered that many of these would
> crash or hang when fed random binary data:
>
> https://www.fuzzingbook.org/html/Fuzzer.html
> ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-revisited.ps
>
> This is a problem because (1) a text utility can be used as a
> general-purpose data manupulation program which is fed binary data (2)
> more importantly: each crash is a potential security hole.
>

People used to be a lot more trusting of unknown data.  It is not a good
idea, and fortunately many programmers have learnt.  Unfortunately, not
/all/ programmers have learned.  (And also those who have learned, are
still fallible humans.)

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Re: Optimization techniques David Brown <david.brown@hesbynett.no> - 2019-04-25 21:58 +0200
  Re: Optimization techniques Kaz Kylheku <847-115-0292@kylheku.com> - 2019-04-26 00:18 +0000
    Re: Optimization techniques David Brown <david.brown@hesbynett.no> - 2019-04-28 23:49 +0200
      Re: Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-04-29 00:31 +0100
        Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-04-29 17:08 +0200
          Re: Optimization techniques and undefined behavior Christian Gollwitzer <auriocus@gmx.de> - 2019-04-29 18:10 +0200
            Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-04-30 14:46 +0200
              Re: Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-01 13:53 +0100
                Re: Optimization techniques and undefined behavior Andy Walker <anw@cuboid.co.uk> - 2019-05-02 11:29 +0100
                Re: Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-03 00:48 +0100
                Re: Optimization techniques and undefined behavior Martin Ward <martin@gkc.org.uk> - 2019-05-03 10:52 +0100
                Re: Optimization techniques and undefined behavior George Neuner <gneuner2@comcast.net> - 2019-05-04 17:44 -0400
                Re: Bounds checking, Optimization techniques and undefined behavior George Neuner <gneuner2@comcast.net> - 2019-05-05 17:10 -0400
                Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-06 08:14 +0200
                Re: Optimization techniques and undefined behavior Gene Wirchenko <genew@telus.net> - 2019-05-11 22:25 -0700
                Re: not a lot of memory, was Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-03 12:45 +0100
                Re: Optimization techniques and undefined behavior Andy Walker <anw@cuboid.co.uk> - 2019-05-03 13:29 +0100
                Re: Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-03 23:10 +0100
                Re: Optimization techniques and undefined behavior Andy Walker <anw@cuboid.co.uk> - 2019-05-04 10:45 +0100
                Re: Bounds checking, Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-05 11:14 +0100
                Re: Bounds checking, Optimization techniques and undefined behavior Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2019-05-05 20:44 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2019-05-06 10:15 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-07 11:04 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior "Nuno Lopes" <nuno.lopes@ist.utl.pt> - 2019-05-07 22:38 +0100
                Re: Bounds checking, Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-08 01:14 +0100
                Re: Bounds checking, Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-08 09:55 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior "Derek M. Jones" <derek@_NOSPAM_knosof.co.uk> - 2019-05-08 19:08 +0100
                Re: Bounds checking, Optimization techniques and undefined behavior Andy Walker <anw@cuboid.co.uk> - 2019-05-08 01:42 +0100
                Re: Bounds checking, Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-08 10:16 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-09 01:15 +0100
                Re: Bounds checking, Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-09 21:56 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-08 10:03 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-09 09:19 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior Kaz Kylheku <847-115-0292@kylheku.com> - 2019-05-10 03:38 +0000
                Re: Bounds checking, Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-08 14:37 +0100
                Re: Bounds checking, Optimization techniques and undefined behavior Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2019-05-06 05:05 -0400
                Re: Bounds checking, Optimization techniques and undefined behavior George Neuner <gneuner2@comcast.net> - 2019-05-05 17:38 -0400
                Re: Bounds checking, Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-06 13:07 +0100
                Re: Bounds checking, Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-07 14:01 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior Andy Walker <anw@cuboid.co.uk> - 2019-05-06 01:15 +0100
                Re: Bounds checking, Optimization techniques and undefined behavior Andy Walker <anw@cuboid.co.uk> - 2019-05-06 14:40 +0100
                Re: Bounds checking, Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-07 15:05 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-08 10:18 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior Jan Ziak <0xe2.0x9a.0x9b@gmail.com> - 2019-05-06 05:39 -0700
                Re: Bounds checking, Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-07 15:42 +0200
                Re: Bounds checking, Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-06 16:32 +0200
                Re: Optimization techniques and undefined behavior George Neuner <gneuner2@comcast.net> - 2019-05-04 17:59 -0400
                Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-02 16:51 +0200
                Re: Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-02 20:04 +0100
                Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-03 17:23 +0200
                Re: Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-03 21:10 +0100
                Re: Optimization techniques and undefined behavior Martin Ward <martin@gkc.org.uk> - 2019-05-06 13:25 +0100
                Re: Optimization techniques and undefined behavior "Derek M. Jones" <derek@_NOSPAM_knosof.co.uk> - 2019-05-06 16:32 +0100
                Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-07 16:03 +0200
                Re: Optimization techniques and undefined behavior Martin Ward <martin@gkc.org.uk> - 2019-05-08 13:16 +0100
                Re: Optimization techniques and undefined behavior George Neuner <gneuner2@comcast.net> - 2019-05-08 15:13 -0400
                Re: Optimization techniques and undefined behavior "Robin Vowels" <robin51@dodo.com.au> - 2019-05-07 01:22 +1000
                Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-07 16:05 +0200
                Re: Optimization techniques and undefined behavior Christian Gollwitzer <auriocus@gmx.de> - 2019-05-02 22:22 +0200
          Re: Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-04-29 18:15 +0100
            Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-04-30 15:48 +0200
              Re: Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-01 12:40 +0100
                Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-02 17:27 +0200
                Re: Optimization techniques and undefined behavior Bart <bc@freeuk.com> - 2019-05-02 18:59 +0100
                Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-07 16:16 +0200
              Re: Optimization techniques and undefined behavior Martin Ward <martin@gkc.org.uk> - 2019-05-02 14:54 +0100
      Re: Optimization techniques and runtime checks Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2019-04-29 22:36 +0200
        Re: Optimization techniques and runtime checks David Brown <david.brown@hesbynett.no> - 2019-05-07 16:29 +0200
          Re: Optimization techniques and runtime checks Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2019-05-08 02:27 +0200
            Re: Optimization techniques and runtime checks David Brown <david.brown@hesbynett.no> - 2019-05-08 10:31 +0200
              Re: Optimization techniques and runtime checks Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2019-05-08 22:50 +0200
              Re: Optimization techniques and runtime checks "Robin Vowels" <robin51@dodo.com.au> - 2019-05-11 19:26 +1000
              Re: Optimization techniques and runtime checks Gene Wirchenko <genew@telus.net> - 2019-05-11 22:43 -0700
                Re: Optimization techniques and runtime checks David Brown <david.brown@hesbynett.no> - 2019-05-12 20:17 +0200
          Re: Optimization techniques and runtime checks Bart <bc@freeuk.com> - 2019-05-08 14:58 +0100
            Re: Optimization techniques and runtime checks David Brown <david.brown@hesbynett.no> - 2019-05-08 23:02 +0200
              Re: Optimization techniques and runtime checks Bart <bc@freeuk.com> - 2019-05-09 18:28 +0100
                Re: Optimization techniques and runtime checks David Brown <david.brown@hesbynett.no> - 2019-05-09 22:07 +0200
      Re: Optimization techniques Gene Wirchenko <genew@telus.net> - 2019-04-30 18:24 -0700
        Re: Optimization techniques David Brown <david.brown@hesbynett.no> - 2019-05-01 09:20 +0200
          Re: Optimization techniques Kaz Kylheku <847-115-0292@kylheku.com> - 2019-05-02 17:40 +0000
          Re: Optimization techniques and error detection Gene Wirchenko <genew@telus.net> - 2019-05-03 10:16 -0700
          Re: Optimization techniques "Robin Vowels" <robin51@dodo.com.au> - 2019-05-07 01:42 +1000
  Re: Optimization techniques Kaz Kylheku <847-115-0292@kylheku.com> - 2019-04-26 02:26 +0000
    Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-04-29 00:12 +0200
      Re: Optimization techniques and undefined behavior Kaz Kylheku <847-115-0292@kylheku.com> - 2019-05-02 17:18 +0000
        Re: Optimization techniques and undefined behavior David Brown <david.brown@hesbynett.no> - 2019-05-07 16:38 +0200

csiph-web