Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.std.c > #1626 > unrolled thread

Second draft, "Safe arrays and pointers for C"

Back to article view | Back to comp.std.c

Contents

  Second draft, "Safe arrays and pointers for C" John Nagle <nagle@animats.com> - 2012-08-30 12:48 -0700
    Re: Second draft, "Safe arrays and pointers for C" pacman@kosh.dhis.org (Alan Curry) - 2012-08-31 22:17 +0000
      Re: Second draft, "Safe arrays and pointers for C" John Nagle <nagle@animats.com> - 2012-08-31 17:26 -0700
        Re: Second draft, "Safe arrays and pointers for C" Hans-Bernhard Bröker <HBBroeker@t-online.de> - 2012-09-02 22:20 +0200
          Re: Second draft, "Safe arrays and pointers for C" BGB <cr88192@hotmail.com> - 2012-09-03 10:55 -0500
            Re: Second draft, "Safe arrays and pointers for C" Ike Naar <ike@iceland.freeshell.org> - 2012-09-03 16:39 +0000
              Re: Second draft, "Safe arrays and pointers for C" BGB <cr88192@hotmail.com> - 2012-09-03 13:18 -0500
    Re: Second draft, "Safe arrays and pointers for C" Jens Gustedt <jens.gustedt@loria.fr> - 2012-09-02 10:52 +0200
    Re: Second draft, "Safe arrays and pointers for C" Per <per.mathisen@gmail.com> - 2012-09-28 06:49 -0700
      Re: Second draft, "Safe arrays and pointers for C" John Nagle <nagle@animats.com> - 2012-09-28 11:36 -0700
        Re: Second draft, "Safe arrays and pointers for C" Per <per.mathisen@gmail.com> - 2012-10-01 09:38 -0700
          Re: Second draft, "Safe arrays and pointers for C" John Nagle <nagle@animats.com> - 2012-10-01 11:38 -0700
        Re: Second draft, "Safe arrays and pointers for C" Shao Miller <sha0.miller@gmail.com> - 2012-12-17 09:11 -0500
          Re: Second draft, "Safe arrays and pointers for C" Keith Thompson <kst-u@mib.org> - 2012-12-17 10:18 -0800
            Re: Second draft, "Safe arrays and pointers for C" Shao Miller <sha0.miller@gmail.com> - 2012-12-17 18:24 -0500
              Re: Second draft, "Safe arrays and pointers for C" Shao Miller <sha0.miller@gmail.com> - 2012-12-17 18:32 -0500
              Re: Second draft, "Safe arrays and pointers for C" Keith Thompson <kst-u@mib.org> - 2012-12-17 15:44 -0800

#1626 — Second draft, "Safe arrays and pointers for C"

In response to all the comments above under "Initial draft proposal:
"Safe arrays and pointers for C", I have a new revision for
discussion.

http://www.animats.com/papers/languages/safearraysforc41.pdf

Major changes:

- It's made clearer that non-strict code can call strict code
  and vice versa, and that function prototypes can be written
  in strict style and called from both strict code (with
  checking) and non-strict code (without checking).  Thus,
  there's a clear migration path to strict code - start
  with the low level libraries and work up.

- The proposed generic functions have been removed. Instead,
  it is proposed to import the "decltype" feature from C++.
  This allows writing the generic functions needed for
  making "malloc" allocation type-safe as C macros.

- The proposal is written relative to N1570.

- Run time subscript and size checking action is
  as specified in N1570, Annex K.

- The relationship with C variable length arrays is discussed.

- The problem of forward-referenced formal parameters
  (where the declaration of an array is followed by
  a parameter with its size) is addressed, using the
  GCC forward declaration extension.

- The restrictions on array length expressions needed
  for consistency between call and definition are made
  explicit.

- Implications for this feature in C++ are briefly discussed,
  the intent being to avoid divergence from C++.  The issue of
  how C++ style "name mangling" and variable length array
  parameters could coexist is addressed.  This
  proposal is now potentially compatible with C++ as
  well as C.

Please comment, as before.  I would particularly like to hear
from people involved with GCC as to how difficult a job it
would be to implement this in GCC as an extension.  Then
we could get some experience with this approach.

				John Nagle

[toc] | [next] | [standalone]

#1634

Usenet followup with a PDF as quoted text sure is difficult. Bad formatting
in the quoted material shall be blamed on pdftotext, iconv, and the bastards
what invented PDF in the first place. I fixed it up just enough to be
readable.

The PDF allegedly says:
>Safe arrays and pointers for C through compatible additions to the language
>John Nagle Discussion draft, round 2- August/September, 2012
>
>Introduction
>Buffer overflows continue to plague C and C++ programs.
                                                        ^
                                                        |
[citation needed]---------------------------------------/

Seriously, got any statistics to support this? What's the percentage of bugs
(or percentage of security holes in particular) that are buffer overflows?
Has this percentage gone up or down over the last 5,10,15,20 years? Would a
stricter language actually have prevented any recent buffer overflow bugs, or
are they caused by programmers failing to use strict compiler modes and other
static analysis tools that are already available for C as it is?

With words like "plague" and no actual numbers, the whole basis for the
proposal looks like a personal grudge.

>Features imported from C++11
>o o o References auto decltype

I don't do C++. Is decltype just a respelling of typeof?

I do know C++ reference syntax and it's stupid. Given this declaration:

int i, j[10], k[3][3], *p, **pp, foo(int), &r;

All of these expressions have type int:

  i, j[i], k[i][i], *p, *pp, foo(i)

And look who's left out: the &r which doesn't mean what it says.

Importing that misdesigned garbage into C is an insult.

>
>Unions may not contain pointer or reference types, and all parts of a union
>must have the same length.

So unions just became useless...

>
>To be modified by pointer arithmetic, a pointer variable must be initialized
>at declaration to an element of an array. Subscripting a pointer is allowed,
>but the result is read-only ("const").

So after all this effort, unintended read accesses via buffer overflow are
still allowed?

>Fixed-length arrays in C and C++
[...]
>This is valid C++ today. C++ already supports the concept of a reference to
>an array. (Arrays of references are prohibited by the C++ standard, but
>references to arrays are not.) Where the compiler knows the length of an
>array, we can acces that informatio via sizeof. This is convenient, but only

Typos (acces, informatio)

>
>When a known-length array is defined or passed as a parameter, its length is
>the value of the dimension expression evaluated at initialization time. This
>follows the usage for C99 variable length array parameters. This feature
>allows a function such as the following:
>void copybyref(size_t n, int (&a)[n], const int (&b)[n]) {
>  for (int i = 0; i < lengthof(a); i++) {
>    a[i] = b[i];
>  }
>}

There is existing syntax (since C99) that I think you're overlooking for the
prototype:

void copybyref(size_t n, int a[static n], const int b[static n])

>If the programmer calls "copybyref" as follows:
>copybyref(lengthof b0, b0, a0);
>The checks become
>lengthof(b0) == lengthof(b0)
>engthof(a0) == lengthof(b0)

Typo (engthof)

>For example, the classic UNIX read call would now be expressed as:
>int read(size_t n; int fd, void_space(&buf)[n], size_t n);
>(The type void_space is discussed below; it's simply a type with a size of
>1, usable for arrays, for which the bytes have no predetermined meaning.
>Think of it as void* with length. The initial "size_t n;" is a forward
>declaration, discussed below.)

void_space is like a hybrid between a void and a char... I think I'd call it
"void char" (it fits in with "signed char" and "unsigned char" as another
variant of the basic 1-byte type). And typedef it as vchar or voidc.

>Worth noting is that the function declaration of read above is, in
>non-strict mode, compatible with existing code which passes buf as a pointer
>to a char.

What about existing code which passes a pointer to a struct, and n is the
sizeof the struct (or maybe a multiple of the sizeof the struct)? Do all
pointers convert to void_space &[] as easily as they convert to void *?

>A struct with an array as the last component
>An especially useful idiom is a structure with an array as the last
>component. C supports this now, but the size of the array is always
>undefined, expressed with [ ] . This proposal allows such arrays to be of
>known size. The array, and the structure, then have a known size. and A
>simple example is
>typedef struct msgitem {
>  const size_t len;
>  char itemvalue[len];
>};
>struct msgitem firstmsg = { 100, 0 }; // empty msgitem, size 100

What's that typedef doing there?

This looks nice, but I had an idea for an explicit syntax marker to show that
an earlier struct element is being referred to. The designated initializer
syntax.

struct msgitem {
  const size_t len;
  char itemvalue[.len];
};

I'm not sure this is better.

The const brings up an important point too. Obviously it's meant to prevent
the len being changed after the struct is allocated. Would you disallow the
declaration without the const?

>
>Associating length with an array pointer in a structure
>Alternatively, a user might want a structure which carries both the array
>and its size. The pointer-based form is typedef str { size_t len, char*
>data}; New reference form: typedef str { size_t len, char (&data)[len]); ...
>str s1 = {n, MALLOC(char, n) }; // structure initialization The key point
>here is that char (&data)[len] is a reference to a string of known length.
>As in the previous example, len, is an element in the same structure, and
>ties the length of the array to a value which can be found from the
>structure. Multiple arrays may be present in the same structure. However,
>all such arrays must be allocated during structure initialization, because a
>reference cannot be null.

Here I think you're reaching too far. You can't just decree that a bunch of
dynamic allocations succeed or fail as some kind of atomic group. You have to
allow for cleanup after the first one succeeds and the second one fails. And
the more you try to hide that cleanup, the less like C your language becomes.

>From Numerical Recipes in C, Gauss-Jordan elimination:
>Original function declaration:
>void gaussj(float **a, int n, float **b, int m);
>New form:
>void gaussj(int m; float (a&)[n][n], int n, float (&b)[n][m], int m);

Are those meant to be ABI-equivalent? I don't see how they can be. The first
one implies that there is actually an array of pointers to float somewhere in
memory, and that the full two-dimensional array of floats doesn't have to be
contiguous.

-- 
Alan Curry

[toc] | [prev] | [next] | [standalone]

#1635

On 8/31/2012 3:17 PM, Alan Curry wrote:
>> Buffer overflows continue to plague C and C++ programs.
>                                                         ^
>                                                         |
> [citation needed]---------------------------------------/
> 
> Seriously, got any statistics to support this? What's the percentage of bugs
> (or percentage of security holes in particular) that are buffer overflows?

   CERT advisories for newly discovered buffer overflows in C programs
continue to come out. Analysis of the CERT database might be helpful here.

> With words like "plague" and no actual numbers, the whole basis for the
> proposal looks like a personal grudge.

   Yes.  I don't like programs crashing.

>> Features imported from C++11
>> o o o References auto decltype
> 
> I don't do C++. Is decltype just a respelling of typeof?

   Essentially, yes.  "decltype" is standard C++11. "typeof" is a
nonstandard GCC extension.

> I do know C++ reference syntax and it's stupid. 

   Sounds like a personal grudge.

>> Unions may not contain pointer or reference types, and all parts of a union
>> must have the same length.
> 
> So unions just became useless...

   Not really.  The main use case for unions is for data read
in from an external source, often a network.  Such data should
not contain pointers or references.

>> To be modified by pointer arithmetic, a pointer variable must be initialized
>> at declaration to an element of an array. Subscripting a pointer is allowed,
>> but the result is read-only ("const").
> 
> So after all this effort, unintended read accesses via buffer overflow are
> still allowed?

   Tightening down the screws further on that would alienate too many C
programmers.  Consider the common idiom

	const char* msg[] =
	{"First message", "second message", "third message", 0 }

Or even
	int main(int argc, const char* argv[]);

Most security attacks require write access.

It might be possible to tighten this down more by restricting such
unchecked access to type char (and wchar_t perhaps) only.  This
is a common idiom for strings, but not that common otherwise.

> 
>> Fixed-length arrays in C and C++
> [...]
>> This is valid C++ today. C++ already supports the concept of a reference to
>> an array. (Arrays of references are prohibited by the C++ standard, but
>> references to arrays are not.) Where the compiler knows the length of an
>> array, we can acces that informatio via sizeof. This is convenient, but only
> 
> Typos (acces, informatio)

   Will fix.

>> When a known-length array is defined or passed as a parameter, its length is
>> the value of the dimension expression evaluated at initialization time. This
>> follows the usage for C99 variable length array parameters. This feature
>> allows a function such as the following:
>> void copybyref(size_t n, int (&a)[n], const int (&b)[n]) {
>>  for (int i = 0; i < lengthof(a); i++) {
>>    a[i] = b[i];
>>  }
>> }
> 
> There is existing syntax (since C99) that I think you're overlooking for the
> prototype:
> 
> void copybyref(size_t n, int a[static n], const int b[static n])

   This was discussed at length back at
"Initial draft proposal: "Safe arrays and pointers for C"".
C99 VLA parameters are delivered to the called function as
a pointer, with the length of the first dimension lost.
Changing that might break some existing program.

    C99 VLA parameters seem to be on the way out, anyway.
In C11, they're an optional feature.  Microsoft refused
to implement them.  And they don't play well with C++
because they weren't designed with function overloading
in mind.

   There's some discussion in the current rev
of the paper on how to make C++ function overloading play
well with these reference parameters, but it's not fully
developed.  That's beyond the scope of this proposal,
but it should be possible to prevent unnecessary divergence
between C and C++.  (Quick note on name mangling: evaluate
all constants in array length expressions down to integers,
rename function params in some positional way like $1, $2,
etc. construct a string which represents the expression
for name matching purposes, and insert that into the
mangled name string at the appropriate point.)


> 
>> If the programmer calls "copybyref" as follows:
>> copybyref(lengthof b0, b0, a0);
>> The checks become
>> lengthof(b0) == lengthof(b0)
>> engthof(a0) == lengthof(b0)
> 
> Typo (engthof)

   Will fix.

>> For example, the classic UNIX read call would now be expressed as:
>> int read(size_t n; int fd, void_space(&buf)[n], size_t n);
>> (The type void_space is discussed below; it's simply a type with a size of
>> 1, usable for arrays, for which the bytes have no predetermined meaning.
>> Think of it as void* with length. The initial "size_t n;" is a forward
>> declaration, discussed below.)
> 
> void_space is like a hybrid between a void and a char... I think I'd call it
> "void char" (it fits in with "signed char" and "unsigned char" as another
> variant of the basic 1-byte type). And typedef it as vchar or voidc.

    That might be a good idea.  Would "void char" as a type break
anything?  Need advice from a syntax expert here.
> 
>> Worth noting is that the function declaration of read above is, in
>> non-strict mode, compatible with existing code which passes buf as a pointer
>> to a char.
> 
> What about existing code which passes a pointer to a struct, and n is the
> sizeof the struct (or maybe a multiple of the sizeof the struct)? Do all
> pointers convert to void_space &[] as easily as they convert to void *?

    If the size matches, yes.  That's the point of "void_space".  Type
matching works like "void", but size works like "char".

>> A struct with an array as the last component
>> An especially useful idiom is a structure with an array as the last
>> component. C supports this now, but the size of the array is always
>> undefined, expressed with [ ] . This proposal allows such arrays to be of
>> known size. The array, and the structure, then have a known size. and A
>> simple example is
>> typedef struct msgitem {
>>  const size_t len;
>>  char itemvalue[len];
>> };
>> struct msgitem firstmsg = { 100, 0 }; // empty msgitem, size 100
> 
> What's that typedef doing there?

    Will fix.

> This looks nice, but I had an idea for an explicit syntax marker to show that
> an earlier struct element is being referred to. The designated initializer
> syntax.
> 
> struct msgitem {
>   const size_t len;
>   char itemvalue[.len];
> };
> 
> I'm not sure this is better.

   The leading "." seems strange.  I'd just start the search for "len"
with the fields of the current struct, then any outer struct, then outer
scopes. Anything found in an outer scope has to be a compile-time
constant.

> 
> The const brings up an important point too. Obviously it's meant to prevent
> the len being changed after the struct is allocated. Would you disallow the
> declaration without the const?

   Yes. You can replace the entire struct by assignment, though.

>> Associating length with an array pointer in a structure
>> Alternatively, a user might want a structure which carries both the array
>> and its size. The pointer-based form is typedef str { size_t len, char*
>> data}; New reference form: typedef str { size_t len, char (&data)[len]); ...
>> str s1 = {n, MALLOC(char, n) }; // structure initialization The key point
>> here is that char (&data)[len] is a reference to a string of known length.
>> As in the previous example, len, is an element in the same structure, and
>> ties the length of the array to a value which can be found from the
>> structure. Multiple arrays may be present in the same structure. However,
>> all such arrays must be allocated during structure initialization, because a
>> reference cannot be null.
> 
> Here I think you're reaching too far. You can't just decree that a bunch of
> dynamic allocations succeed or fail as some kind of atomic group. You have to
> allow for cleanup after the first one succeeds and the second one fails. And
> the more you try to hide that cleanup, the less like C your language becomes.

    I'm merely suggesting that an allocation function be provided
that calls some global function on an out of memory condition. It's
still possible to do it the hard way, with recovery from each failed
allocation.  But a fail-on-allocate feature could be useful.  It's
not going to clean up without memory leaks, as can be done in C++.
I'm just suggesting that as a convenience feature.
> 
>>From Numerical Recipes in C, Gauss-Jordan elimination:
>> Original function declaration:
>> void gaussj(float **a, int n, float **b, int m);
>> New form:
>> void gaussj(int m; float (a&)[n][n], int n, float (&b)[n][m], int m);
> 
> Are those meant to be ABI-equivalent? I don't see how they can be. The first
> one implies that there is actually an array of pointers to float somewhere in
> memory, and that the full two-dimensional array of floats doesn't have to be
> contiguous.

   Good point.  Can't quite do that one compatibly.  Most arrays
passed that way are in fact contiguous, but they don't have to be.

				John Nagle

[toc] | [prev] | [next] | [standalone]

#1650

On 01.09.2012 02:26, John Nagle wrote:

>     Not really.  The main use case for unions is for data read
> in from an external source, often a network.

No, it's not, because the vast majority of usages of unions for that 
purpose would automatically cause undefined behaviour.

>     Tightening down the screws further on that would alienate too many C
> programmers.  Consider the common idiom
>
> 	const char* msg[] =
> 	{"First message", "second message", "third message", 0 }

And that would run afoul of reading beyond the end of the array ... how?

> Or even
> 	int main(int argc, const char* argv[]);
>
> Most security attacks require write access.

Let's hold it right here for a moment.  So what is it you're trying to 
avoid: attacks or program crashes?  The distinction is quite important, 
because actual security attacks need write access a whole lot less than 
crashing programs would.  You can expose secret data by random reading, 
but it's a whole lot harder to crash a program that way.

>      C99 VLA parameters seem to be on the way out, anyway.

That hasn't become any less incorrect since the last time round.

[toc] | [prev] | [next] | [standalone]

#1655

On 9/2/2012 3:20 PM, Hans-Bernhard Bröker wrote:
> On 01.09.2012 02:26, John Nagle wrote:
>
>>     Not really.  The main use case for unions is for data read
>> in from an external source, often a network.
>
> No, it's not, because the vast majority of usages of unions for that
> purpose would automatically cause undefined behaviour.
>

yeah, and also most cases I have seen unions used for have little to do 
with network data.


>>     Tightening down the screws further on that would alienate too many C
>> programmers.  Consider the common idiom
>>
>>     const char* msg[] =
>>     {"First message", "second message", "third message", 0 }
>

(ironically, the above will probably only work without issue in C++, and 
in C will usually generate a message something like "warning: int to 
pointer conversion without a cast" or similar... this means the array 
needs to be terminated with either NULL or "((void *)0)" or "((char 
*)0)" or similar to avoid the compiler complaining about it).


> And that would run afoul of reading beyond the end of the array ... how?
>

dunno what he was thinking here...

my guess is maybe that unless whatever is used as an index is clamped, 
it may go beyond the end of the array.

but, then, usually in cases like these, a person is using a number that 
they know is an index into the array, so it isn't much of an issue.


>> Or even
>>     int main(int argc, const char* argv[]);
>>
>> Most security attacks require write access.
>
> Let's hold it right here for a moment.  So what is it you're trying to
> avoid: attacks or program crashes?  The distinction is quite important,
> because actual security attacks need write access a whole lot less than
> crashing programs would.  You can expose secret data by random reading,
> but it's a whole lot harder to crash a program that way.
>

yep.


>>      C99 VLA parameters seem to be on the way out, anyway.
>
> That hasn't become any less incorrect since the last time round.

yep.


in my own ideas, usually, a bounds-checked array index or similar would 
be redirected to the runtime, which would attempt to look up the object, 
and then check the types and index (operations such as 
pointer-arithmetic would be bounds-checked as well).

in this case, the compiler+runtime would make provisions for keeping 
track of types and performing bounds-checks and type-checks, and so this 
would usually come at a performance cost (need not be prohibitive, if 
designed well).

in this case, both language and ABI compatibility could be maintained 
(if the physical representation of the data is maintained, generally 
because out-of-band metadata would be used to keep track of things).

note: a design goal is to keep the pointer-lookups reasonably fast. 
usually, this involves a little care in the design of a memory manager, 
but the lookups can often be reduced to an O(1) average case (and an 
O(log n) worst case).

note: by metadata, I am imagining more specialized type information, not 
the use of generic debug information.


the open question would be how to best handle the case of interaction 
with outside code or data, which may provide pointers which lack visible 
type information, requiring either being "lax" (allowing unchecked 
operations in these cases), or "strict" (say, raising a signal or OS 
exception).


not that I expect such an idea would be universally applicable, but 
could be useful for "untrusted" code, for debugging purposes, and code 
which is "generally not CPU bound" (less need to care about performance 
of code when the CPU is mostly sitting around idle).

[toc] | [prev] | [next] | [standalone]

#1656

On 2012-09-03, BGB <cr88192@hotmail.com> wrote:
> On 9/2/2012 3:20 PM, Hans-Bernhard Br?ker wrote:
>> On 01.09.2012 02:26, John Nagle wrote:
>>
>>>     Not really.  The main use case for unions is for data read
>>> in from an external source, often a network.
>>
>> No, it's not, because the vast majority of usages of unions for that
>> purpose would automatically cause undefined behaviour.
>>
>
> yeah, and also most cases I have seen unions used for have little to do 
> with network data.
>
>
>>>     Tightening down the screws further on that would alienate too many C
>>> programmers.  Consider the common idiom
>>>
>>>     const char* msg[] =
>>>     {"First message", "second message", "third message", 0 }
>>
>
> (ironically, the above will probably only work without issue in C++, and 
> in C will usually generate a message something like "warning: int to 
> pointer conversion without a cast" or similar... this means the array 
> needs to be terminated with either NULL or "((void *)0)" or "((char 
> *)0)" or similar to avoid the compiler complaining about it).

In C, 0 is perfectly fine as a char* (it's a null pointer constant).
Have you really seen warnings for this?

[toc] | [prev] | [next] | [standalone]

#1657

On 9/3/2012 11:39 AM, Ike Naar wrote:
> On 2012-09-03, BGB <cr88192@hotmail.com> wrote:
>> On 9/2/2012 3:20 PM, Hans-Bernhard Br?ker wrote:
>>> On 01.09.2012 02:26, John Nagle wrote:
>>>
>>>>      Not really.  The main use case for unions is for data read
>>>> in from an external source, often a network.
>>>
>>> No, it's not, because the vast majority of usages of unions for that
>>> purpose would automatically cause undefined behaviour.
>>>
>>
>> yeah, and also most cases I have seen unions used for have little to do
>> with network data.
>>
>>
>>>>      Tightening down the screws further on that would alienate too many C
>>>> programmers.  Consider the common idiom
>>>>
>>>>      const char* msg[] =
>>>>      {"First message", "second message", "third message", 0 }
>>>
>>
>> (ironically, the above will probably only work without issue in C++, and
>> in C will usually generate a message something like "warning: int to
>> pointer conversion without a cast" or similar... this means the array
>> needs to be terminated with either NULL or "((void *)0)" or "((char
>> *)0)" or similar to avoid the compiler complaining about it).
>
> In C, 0 is perfectly fine as a char* (it's a null pointer constant).
> Have you really seen warnings for this?
>

I remembered seeing them in the past.

testing again, I don't seem to be getting a warning with either MSVC or 
GCC, so maybe I was wrong or misremembering?... (or it was a 
compiler-version specific issue with a past version of GCC or similar?...)

at the moment, there is no issue with a constant value of 0, but if the 
value is anything other than 0, then there is a message.


fair enough, 0 seems to work...

[toc] | [prev] | [next] | [standalone]

#1648

Am 30.08.2012 21:48, schrieb John Nagle:
> In response to all the comments above under "Initial draft proposal:
> Safe arrays and pointers for C", I have a new revision for
> discussion.
> 
> http://www.animats.com/papers/languages/safearraysforc41.pdf
> 
> Major changes:
> 
> - It's made clearer that non-strict code can call strict code
>   and vice versa, and that function prototypes can be written
>   in strict style and called from both strict code (with
>   checking) and non-strict code (without checking).  Thus,
>   there's a clear migration path to strict code - start
>   with the low level libraries and work up.
> 
> - The proposed generic functions have been removed. Instead,
>   it is proposed to import the "decltype" feature from C++.
>   This allows writing the generic functions needed for
>   making "malloc" allocation type-safe as C macros.
> 
> - The proposal is written relative to N1570.
> 
> - Run time subscript and size checking action is
>   as specified in N1570, Annex K.
> 
> - The relationship with C variable length arrays is discussed.
> 
> - The problem of forward-referenced formal parameters
>   (where the declaration of an array is followed by
>   a parameter with its size) is addressed, using the
>   GCC forward declaration extension.
> 
> - The restrictions on array length expressions needed
>   for consistency between call and definition are made
>   explicit.
> 
> - Implications for this feature in C++ are briefly discussed,
>   the intent being to avoid divergence from C++.  The issue of
>   how C++ style "name mangling" and variable length array
>   parameters could coexist is addressed.  This
>   proposal is now potentially compatible with C++ as
>   well as C.
> 
> Please comment, as before.  I would particularly like to hear
> from people involved with GCC as to how difficult a job it
> would be to implement this in GCC as an extension.  Then
> we could get some experience with this approach.

I see that you worked well through the objections that had been
presented, great. Your new proposal is certainly much closer to how
things are usually seen in C.

I still think that you push things too far. I completely understand
your motivation, that you want to have usable and doable version in
one go, but I still think that it would be far easier acceptable if
you'd present a series of additions to the standard that each would be
relatively easy to follow. (A serious of patches, if you like).

In the meantime, I have implement part of what you proposed as macros
for C. P99 now has a set of macros with prefix "P99_CA_" (CA for check
arguments) that allow to "automatically" wrap function calls such that
the arguments are checked for their size. This all works with pointers
to VLA, not references since they simply don't exist. And as always
for such macro approaches the declaration/definition part is not too
pretty, but the usage side looks just a normal function call and the
checking is done behind the scenes.

The only assumption for VM types as parameters that has to be done is
that any function prototypes that uses them have exactly the same size
expressions as their definitions. (Should be obvious that you have to
expose the size expressions to the caller.)

All this to show that parameter size checking within the idiom of
pointers to VLA is already possible. The usage pattern of (*A)[i][j]
is obviously not too comfortable, so adding simple reference in a
later stage would probably a good move.

As I also see from your proposal, there is a loophole in the existing
standard that is opened by "malloc" and friends. The language
currently simply has not much possibilities to assert that an untyped
pointer represents an object of a certain size. So there would be need
for an extension of the language that makes some guarantees. I would
be more conservative than you, and just asking to implement a
macro/function/operator that guarantees allocation and returns the
proper type. (Failure of such an operator should be handle as a
runtime constraint violation.)

I am currently working that P99_MALLOC(X) (currently just expands to
malloc(sizeof(X))) returns a pointer to the correct type on platforms
where such a thing is possible.

Jens

[toc] | [prev] | [next] | [standalone]

#1679

I like the intention behind this proposal. Being able to pass on the size information of an array, and read it out later, would greatly enhance C and make horrors like TR 24731-1 (the Microsoft safe-c proposal) obsolete. However, I think it would be far nicer and play less havoc with C to do this with the addition of a new fat pointer type.

Two years ago, I proposed a way of doing this:
https://groups.google.com/d/msg/comp.std.c/ZXPkW93gSow/sq2jouycc-sJ

I still think this proposal has some merit.

Yours,
Per

[toc] | [prev] | [next] | [standalone]

#1680

On 9/28/2012 6:49 AM, Per wrote:
> I like the intention behind this proposal. Being able to pass on the
> size information of an array, and read it out later, would greatly
> enhance C and make horrors like TR 24731-1 (the Microsoft safe-c
> proposal) obsolete. However, I think it would be far nicer and play
> less havoc with C to do this with the addition of a new fat pointer
> type.

   That's been tried a few times.  There's a GCC extension for it,
and it was tried in the Safe C compiler and in Cyclone.  It
didn't catch on.   Pointer overhead increases substantially,
and you can't call existing libraries.

   I have a new version of my paper on "strict mode" out.

http://www.animats.com/papers/languages/safearraysforc43.pdf

This deals with most of the criticisms to date, and firms
up the type checking rules.  A key point is that the rules
for non-strict code calling strict code and vice versa are
made clear.  A basic goal is to make it possible to have
strict libraries and APIs, callable from non-strict or
strict code.  This allows graceful conversion of existing
code, one translation unit at a time.  A big fraction of
security related buffer overflow exploits involve size
issues at API boundaries.  So this addresses a major real-world issue.

   Please take a look and see if you can find any more problems.
I'm looking at this as a possible extension to GCC, which already
has some of the extensions I'm proposing.

				John Nagle

[toc] | [prev] | [next] | [standalone]

#1685

20:36:37 UTC+2 28. september 2012 John Nagle wrote:
> On 9/28/2012 6:49 AM, Per wrote:
>    That's been tried a few times.  There's a GCC extension for it,
> and it was tried in the Safe C compiler and in Cyclone.  It
> didn't catch on.   Pointer overhead increases substantially,
> and you can't call existing libraries.

That is not accurate. What I proposed was not converting unsafe pointers in general to fat pointers, which I am aware of has been a failure and has non-trivial overhead, but instead adding fat pointers as an additional option for programmers. This allows you to use all existing libraries without modification, and where there is overhead, or you just feel like it, you can keep using unsafe pointers as before.

Your proposal keeps accreting features, just like my own ideas did. There are so many nice things that you can do once you have more advanced pointers! But I think that the only way to get anywhere is to keep it dead simple.

What you really want is a way to keep known lengths around while being tightly coupled with the arrays they describe. Your function parameter syntax is interesting. I'm uneasy with your suggestion to leave it up to the compiler to generate the appropriate temporaries to maintain this information, though. I think C is a language where such things are meant to be explicit (as opposed to C++).

I think I see a way to do this in a really nice way now, inspired by your proposal. I will put that in a separate thread shortly.

Yours,
Per

[toc] | [prev] | [next] | [standalone]

#1686

On 10/1/2012 9:38 AM, Per wrote:
> 20:36:37 UTC+2 28. september 2012 John Nagle wrote:
>> On 9/28/2012 6:49 AM, Per wrote: That's been tried a few times.
>> There's a GCC extension for it, and it was tried in the Safe C
>> compiler and in Cyclone.  It didn't catch on.   Pointer overhead
>> increases substantially, and you can't call existing libraries.
> 
> That is not accurate. What I proposed was not converting unsafe
> pointers in general to fat pointers, which I am aware of has been a
> failure and has non-trivial overhead, but instead adding fat pointers
> as an additional option for programmers. This allows you to use all
> existing libraries without modification, and where there is overhead,
> or you just feel like it, you can keep using unsafe pointers as
> before.

   A "just feel like it" approach will probably not reduce errors
substantially.  Stronger assurances of pointer safety are required
to reduce buffer overflow vulnerabilities.
> 
> Your proposal keeps accreting features, just like my own ideas did.
> There are so many nice things that you can do once you have more
> advanced pointers! But I think that the only way to get anywhere is
> to keep it dead simple.

    As you can see from the various drafts, I've thrown
some features out.  The "generic functions" I previously proposed
to deal with safe handling of "malloc" are gone, replaced with
defines that encapsulate casts.  Most of what I'm proposing is
borrowed from the non-OOP parts of C++.  This keeps things close to
the spirit of C.

    I'm also trying to make these features C++ compatible.
The rejection of C99 VLAs by Microsoft reflects the fact that
they can't be fitted into C++.  What I'm proposing can be
fitted into C++ overloading and name mangling if desired, so
it doesn't take C in a direction that C++ can't go.

> What you really want is a way to keep known lengths around while
> being tightly coupled with the arrays they describe. Your function
> parameter syntax is interesting. I'm uneasy with your suggestion to
> leave it up to the compiler to generate the appropriate temporaries
> to maintain this information, though. I think C is a language where
> such things are meant to be explicit (as opposed to C++).

    Compilers generate local temporaries all the time, just to
compile expressions.  I agree that it's best to avoid associating
additional data with user-declared variables.  User-declared
variables in C should have the machine representation that users
expect them to have.  Much code depends on that.  A program can't
explicitly access local temporaries, though, so it doesn't matter
how the compiler handles them.  They can be in a register or on the
stack, for example, and the programmer can't tell. An optimizing
compiler may precompute an expression, hoist it out of a loop,
and store it in a temporary that lives for the duration of the loop.
So C already has long-lived internal local temporaries.

    The function parameter syntax is basically C++ reference syntax,
extended in the way C99 VLAs were extended.  We need some way to talk
about an array as an array, and a C++ reference is a well-understood
way to do that.

    That's the whole point of this proposal - programs
should describe arrays as arrays, rather than reducing them to raw
pointers.  The hard part is doing this in a way that's backwards
compatible.

> I think I see a way to do this in a really nice way now, inspired by
> your proposal. I will put that in a separate thread shortly.

     OK.

					John Nagle

[toc] | [prev] | [next] | [standalone]

#1740

On August 30th, 2012, at 12:23:29, John Nagle published:
> Safe arrays and pointers for C through compatible additions to the language

Some feed-back, if you care:

I don't really care for importing references from C++11.  C has long 
passed by value (with the exception of arrays [which itself has an 
exception in C89/C90]).

Importing references might mean losing the ability to advise a 
C-newcomer with a keyboard-shortcut that does "C passes by value."

You had a C++ code example:

>   int arr100[100]; // an array of 100 ints
>   arr100& arr100r; // ref to array of 100 ints

My C++ is too rusty to know if there is an error here, or not.  If it's 
really the case that you can do:

   sometype someobj;
   someobj& someobjref;

Then that'd certainly make for some interesting syntactical differences, 
in C!  Does 'someobj&' become a 'type-name'?  If so, at what stages is 
it an incomplete or complete object type?  Can the same be done with 
function identifiers?  Can it be confused for something like:

   (void) (someobj & undeclared_identifier);

?  Is the type of 'someobjref' 'sometype *'?  If so, can its value be 
reset, or is it 'sometype * const'?  How might you declare a reference 
like this as part of a 'struct' or 'union' type definition?

>   void copybyref(size_t n, int (&a)[n], const int (&b)[n])
>   {
>       for (int i = 0; i < lengthof(a); i++)
>       {   a[i] = b[i]; }
>   }

Before jumping right into using '&' in the parameters, I think it'd ease 
my understanding of the proposal if I didn't have to read all of C++11 
just to understand what "references" are, and how they work.  That is, 
the proposal is for C, but the easy import of references from C++11 mean 
that the proposal-reader (probably interested in C) is at a disadvantage 
if they're not familiar with C++11's references.

Considering this particular code example, it must be leading up to an 
eventual alternative, as 'n' appears to make 'lengthof(a)' unnecessary.

It might be interesting if there was a way to just have 'a' and 'b' 
parameters, and to have some mechanism that is independent of your 
proposal's material to ensure that the types of 'a' and 'b' are 
compatible (or have the same size, or whatever) at the call-site.  A 
macro with '_Generic' might help, but perhaps something a little less 
cluttered.

> int read(size_t n; int fd, void_space(&buf)[n], size_t n);

This must still be leading up to something, as 'n' is still being passed...

With the proposed forward declaration idea, can initializers go in 
there?  What about declaring identifiers just for fun but that aren't 
used by the parameters?  Can linkage and storage be specified in those 
forward declarations?  What about declaring functions and enums in there?

>   int b[100]; // array of 100 ints
>   ...
>   int (&rb)[lengthof(b)] = b; // ref to array b
>
> Assignments to references are subject to type checking

Did you mean "initializers" or did you really mean assignments?

> Conversion from anything which can be NULL to a reference implies a run-time constraint
> check

That C++ references offer any protection against having "null 
references" has been discussed in comp.lang.c and, I hope, dismissed. 
You might be interested in "Null References in C++" at this "Code 
snippets" page: http://www.iso-9899.info/wiki/Code_snippets

I recognize that you are suggesting this "strict mode" as a value-adding 
mode, so I can appreciate that the "code snippets" example might also be 
caught by some C++ implementations employing a similarly valuable mode.

> To be modified by pointer arithmetic, a pointer variable must be initialized at declaration to an
> element of an array

(Presumably "to an element" should be "to point to an element".)

So not to the result of an allocation function, nor to a 'sometype * 
const' that was itself initialized to point to an element of an array, 
right?

> The only allowed values for the pointer variable used in pointer arithmetic are ones which point
> to an element of the array to which it was originally assigned

Did you mean "initialized to point to" or did you really mean "assigned"?

>   int asize = 100; // size wanted
>   int atab[asize]; // declare known-length array
>   asize = 200; // this does not resize the array
>   printf(“Length of atab is %d\n”, lengthof(atab)); // prints 100

'sizeof' does this for VLAs already, so this instance of 'lengthof' can 
be implemented by the usual macro idiom.

>   typedef struct msgitem {
>       const size_t len;
>       char itemvalue[len];
>   };
>   struct msgitem firstmsg = { 100, 0 }; // empty msgitem, size 100

Interesting.  Could the bounds for 'itemvalue' be any kind of expression 
involving any of the previous members' values?  If I already had 'len' 
as a declared identifier just before this 'typedef', would the inner 
'len' hide the outer one?  If so, I gather than any implementation 
already offering VLAs as the last member, as an implementation 
extension, would be incompatible?

Time to move onto something else for a while, but it's been an 
interesting read, so far!  Thanks.  I wonder if you could hack away at 
the proposed changes and reformulate as a series of tiny, digestible 
changes that could accumulate some day and result in the realization of 
the whole.

- Shao Miller

[toc] | [prev] | [next] | [standalone]

#1741

Shao Miller <sha0.miller@gmail.com> writes:
> On August 30th, 2012, at 12:23:29, John Nagle published:
>> Safe arrays and pointers for C through compatible additions to the language
>
> Some feed-back, if you care:
>
> I don't really care for importing references from C++11.  C has long 
> passed by value (with the exception of arrays [which itself has an 
> exception in C89/C90]).

Strictly speaking, arrays are not an exception to the C rule that
function arguments are passed by value.  Given

    int arr[N];
    func(arr);

an array is not being passed as an argument; rather, a pointer is
being passed (by value, of course).  The implicit conversion of
the array expression to a pointer is unrelated to the fact that
it appears in a function call; the same conversion happens in
most contexts.

[...]

If C were to add some kind of pass-by-reference mechanism (other than
the existing pass-a-pointer-by-value), it might be useful to require
some distinguishing syntax on the call.  For example, assuming the
addition of a "ref" keyword:

    void increment(ref int n) {
        n ++;
    }

    int x = 42;
    /* increment(x); would be illegal */
    increment(ref x);
    /* now x == 43 */

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
    Will write code for food.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#1742

On 12/17/2012 13:18, Keith Thompson wrote:
> Shao Miller <sha0.miller@gmail.com> writes:
>> On August 30th, 2012, at 12:23:29, John Nagle published:
>>> Safe arrays and pointers for C through compatible additions to the language
>>
>> Some feed-back, if you care:
>>
>> I don't really care for importing references from C++11.  C has long
>> passed by value (with the exception of arrays [which itself has an
>> exception in C89/C90]).
>
> Strictly speaking, arrays are not an exception to the C rule that
> function arguments are passed by value.  Given
>
>      int arr[N];
>      func(arr);
>
> an array is not being passed as an argument; rather, a pointer is
> being passed (by value, of course).  The implicit conversion of
> the array expression to a pointer is unrelated to the fact that
> it appears in a function call; the same conversion happens in
> most contexts.

Yes; thanks.  The omission of ", which aren't passed at all" before the 
space just before the left square bracket was an unhelpful one.

>
> [...]
>
> If C were to add some kind of pass-by-reference mechanism (other than
> the existing pass-a-pointer-by-value), it might be useful to require
> some distinguishing syntax on the call.  For example, assuming the
> addition of a "ref" keyword:
>
>      void increment(ref int n) {
>          n ++;
>      }
>
>      int x = 42;
>      /* increment(x); would be illegal */
>      increment(ref x);
>      /* now x == 43 */
>

While I agree that distinguishing syntax is desirable, I don't quite 
follow what the advantage of that particular example is, over pointers. 
  (Of course, that mightn't've been a demonstration of an advantage, but 
simply of distinguishing syntax.)

What I don't quite understand about the proposal (yet, but there's more 
for me to read) is the benefit.  Since the size of an array (or count of 
its elements) isn't _always_ needed (sentinel values, for example), why 
try to tie it more closely to arrays?  I'll re-emphasize _always_, just 
now; a nice-to-have, _if_ you want it.

It seems that the major issue being addressed is buffer overflow, some 
of which could be caught at translation-time (as a form of type-safety?) 
and some of which could be caught at run-time (as a form of run-time 
type-checking?).  Isn't that what C99's 'static' in an "array" (but 
really pointer) parameter declaration in a function declaration was for?

Perhaps it'll be clearer after reading the rest of the proposal. :)

- Shao Miller

[toc] | [prev] | [next] | [standalone]

#1743

On 12/17/2012 18:24, Shao Miller wrote:
> It seems that the major issue being addressed is buffer overflow

Er, overrun.  Whatever.  (Trying to get into the habit of reserving 
"overflow" for value ranges, rather than item ranges.)

[toc] | [prev] | [next] | [standalone]

#1744

Shao Miller <sha0.miller@gmail.com> writes:
> On 12/17/2012 13:18, Keith Thompson wrote:
[...]
>> If C were to add some kind of pass-by-reference mechanism (other than
>> the existing pass-a-pointer-by-value), it might be useful to require
>> some distinguishing syntax on the call.  For example, assuming the
>> addition of a "ref" keyword:
>>
>>      void increment(ref int n) {
>>          n ++;
>>      }
>>
>>      int x = 42;
>>      /* increment(x); would be illegal */
>>      increment(ref x);
>>      /* now x == 43 */
>>
>
> While I agree that distinguishing syntax is desirable, I don't quite 
> follow what the advantage of that particular example is, over pointers. 
>   (Of course, that mightn't've been a demonstration of an advantage, but 
> simply of distinguishing syntax.)

It's not an advantage over pointers.  It's an advantage over a
hypothetical pass-by-reference mechanism in which you can't tell,
by looking at a call, whether that call might modify the value of
the passed object.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
    Will write code for food.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [standalone]

Back to top | Article view | comp.std.c

csiph-web