Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #99309 > unrolled thread

Why is this outputting alternate blank lines?

Back to article view | Back to comp.lang.c

Contents

  Why is this outputting alternate blank lines? DFS <nospam@dfs.com> - 2016-12-13 23:18 -0500
    Re: Why is this outputting alternate blank lines? Melzzzzz <mel@zzzzz.com> - 2016-12-14 05:30 +0100
    Re: Why is this outputting alternate blank lines? mark.bluemel@gmail.com - 2016-12-14 00:59 -0800
    Re: Why is this outputting alternate blank lines? Richard Heathfield <rjh@cpax.org.uk> - 2016-12-14 10:38 +0000
      Re: Why is this outputting alternate blank lines? DFS <nospam@dfs.com> - 2016-12-14 12:53 -0500
        Re: Why is this outputting alternate blank lines? Richard Heathfield <rjh@cpax.org.uk> - 2016-12-14 18:05 +0000
          Re: Why is this outputting alternate blank lines? Keith Thompson <kst-u@mib.org> - 2016-12-14 11:57 -0800
            Re: Why is this outputting alternate blank lines? Richard Heathfield <rjh@cpax.org.uk> - 2016-12-14 20:05 +0000
              Re: Why is this outputting alternate blank lines? "James R. Kuyper" <jameskuyper@verizon.net> - 2016-12-14 15:21 -0500
              Re: Why is this outputting alternate blank lines? Keith Thompson <kst-u@mib.org> - 2016-12-14 12:28 -0800
                Re: Why is this outputting alternate blank lines? supercat@casperkitty.com - 2016-12-14 13:09 -0800
                Re: Why is this outputting alternate blank lines? Jean-Marc Bourguet <jm@bourguet.org> - 2016-12-14 22:53 +0100
                  Re: Why is this outputting alternate blank lines? <william@wilbur.25thandClement.com> - 2016-12-14 15:47 -0800
              Re: Why is this outputting alternate blank lines? Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2016-12-14 15:41 -0700
                Re: Why is this outputting alternate blank lines? Richard Heathfield <rjh@cpax.org.uk> - 2016-12-14 23:26 +0000
                  Re: Why is this outputting alternate blank lines? Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2016-12-14 17:33 -0700
                    Re: Why is this outputting alternate blank lines? Richard Heathfield <rjh@cpax.org.uk> - 2016-12-15 00:42 +0000
                      Re: Why is this outputting alternate blank lines? supercat@casperkitty.com - 2016-12-14 17:05 -0800
                        Re: Why is this outputting alternate blank lines? Richard Heathfield <rjh@cpax.org.uk> - 2016-12-15 08:21 +0000
                          Re: Why is this outputting alternate blank lines? supercat@casperkitty.com - 2016-12-15 09:15 -0800
                            Re: Why is this outputting alternate blank lines? Richard Heathfield <rjh@cpax.org.uk> - 2016-12-15 17:50 +0000
                              Re: Why is this outputting alternate blank lines? supercat@casperkitty.com - 2016-12-15 12:15 -0800
                                Re: Why is this outputting alternate blank lines? Richard Heathfield <rjh@cpax.org.uk> - 2016-12-15 20:37 +0000
                            Re: Why is this outputting alternate blank lines? raltbos@xs4all.nl (Richard Bos) - 2016-12-15 20:08 +0000
                      Re: Why is this outputting alternate blank lines? Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2016-12-14 20:40 -0700
                        Re: Why is this outputting alternate blank lines? Richard Heathfield <rjh@cpax.org.uk> - 2016-12-15 08:36 +0000
                          Re: Why is this outputting alternate blank lines? Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2016-12-15 17:43 -0700
                            Re: Why is this outputting alternate blank lines? Richard Heathfield <rjh@cpax.org.uk> - 2016-12-16 01:05 +0000
                              Re: Why is this outputting alternate blank lines? Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2016-12-15 18:34 -0700
                  Re: Why is this outputting alternate blank lines? <william@wilbur.25thandClement.com> - 2016-12-15 10:14 -0800
                    Re: Why is this outputting alternate blank lines? raltbos@xs4all.nl (Richard Bos) - 2016-12-15 20:15 +0000
              Re: Why is this outputting alternate blank lines? Tim Rentsch <txr@alumni.caltech.edu> - 2016-12-15 01:51 -0800
                Re: Why is this outputting alternate blank lines? asetofsymbols@gmail.com - 2016-12-15 06:09 -0800
                  Re: Why is this outputting alternate blank lines? Tim Rentsch <txr@alumni.caltech.edu> - 2016-12-15 17:47 -0800
            Re: Why is this outputting alternate blank lines? Keith Thompson <kst-u@mib.org> - 2016-12-14 12:15 -0800
              Re: Why is this outputting alternate blank lines? <william@wilbur.25thandClement.com> - 2016-12-15 10:24 -0800
            Re: overcommit is the plague Noob <root@127.0.0.1> - 2016-12-17 01:17 +0100
        Re: Why is this outputting alternate blank lines? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2016-12-14 20:19 +0000
          Re: Why is this outputting alternate blank lines? DFS <nospam@dfs.com> - 2016-12-14 17:44 -0500
            Re: Why is this outputting alternate blank lines? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2016-12-15 00:21 +0000
              Re: Why is this outputting alternate blank lines? mark.bluemel@gmail.com - 2016-12-15 00:56 -0800
                Re: Why is this outputting alternate blank lines? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2016-12-15 22:08 +0000
        Re: Why is this outputting alternate blank lines? Geoff <geoff@invalid.invalid> - 2016-12-14 13:39 -0800
        Re: Why is this outputting alternate blank lines? Ike Naar <ike@iceland.freeshell.org> - 2016-12-16 07:12 +0000

Page 1 of 3  [1] 2 3  Next page →

#99309 — Why is this outputting alternate blank lines?

Trying to read in a file of date string (in this case 10) and add them 
to an array

input
22-May-1819
01-Jan-1268
03-May-1417
01-Mar-1250
05-Oct-1566
21-Feb-1629
23-Jun-1478
18-Feb-1576
07-Jan-1099
21-Mar-1299


=====================================================
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main()
{
   int i=0,n=0;
   char *datestr[20];  //segfaults unless 2*input lines
   char line[12];
   FILE *fp = fopen("datestrings.txt","r");
   while(fgets(line,sizeof(line),fp)!=NULL) {
     datestr[i]=malloc(sizeof(line));
     strcpy(datestr[i],line);
     i++;n++;
   }
   fclose(fp);
   for (i=0;i<n;i++){
    printf("%s\n", datestr[i]);
   }
}
=====================================================

output
22-May-1819


01-Jan-1268


03-May-1417


01-Mar-1250


05-Oct-1566


21-Feb-1629


23-Jun-1478


18-Feb-1576


07-Jan-1099


21-Mar-1299




Any suggestions on how to fix it, or do it better?

Thanks

[toc] | [next] | [standalone]

#99310

On Tue, 13 Dec 2016 23:18:32 -0500
DFS <nospam@dfs.com> wrote:

> Trying to read in a file of date string (in this case 10) and add
> them to an array
> 
> input
> 22-May-1819
> 01-Jan-1268
> 03-May-1417
> 01-Mar-1250
> 05-Oct-1566
> 21-Feb-1629
> 23-Jun-1478
> 18-Feb-1576
> 07-Jan-1099
> 21-Mar-1299
> 
> 
> =====================================================
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> 
> int main()
> {
>    int i=0,n=0;
>    char *datestr[20];  //segfaults unless 2*input lines
>    char line[12];
>    FILE *fp = fopen("datestrings.txt","r");
>    while(fgets(line,sizeof(line),fp)!=NULL) {
>      datestr[i]=malloc(sizeof(line));
>      strcpy(datestr[i],line);
>      i++;n++;
>    }
>    fclose(fp);
>    for (i=0;i<n;i++){
>     printf("%s\n", datestr[i]);
>    }
> }
> =====================================================
> 
> output
> 22-May-1819
> 
> 
> 01-Jan-1268
> 
> 
> 03-May-1417
> 
> 
> 01-Mar-1250
> 
> 
> 05-Oct-1566
> 
> 
> 21-Feb-1629
> 
> 
> 23-Jun-1478
> 
> 
> 18-Feb-1576
> 
> 
> 07-Jan-1099
> 
> 
> 21-Mar-1299
> 
> 
> 
> 
> Any suggestions on how to fix it, or do it better?
> 
> Thanks
> 
> 

fgets does not discards eol characters and reads maximum size - 1
characters, therefore new line is read as separate array entry.

-- 
press any key to continue or any other to quit...

[toc] | [prev] | [next] | [standalone]

#99316

On Wednesday, 14 December 2016 04:18:40 UTC, DFS  wrote:
> Trying to read in a file of date string (in this case 10) and add them 
> to an array
> 
> input
> 22-May-1819
As Melzzzzz points out, if you read one of these lines into a 12-char array 
with fgets(), it will need to reserve 1 char for the delimiting null.

So the first read will retrieve "22-may-1819" and terminate it.
The second read will retrieve the newline character and terminate it.
etc...

That's why your array needs twice the number of entries you were expecting.

[toc] | [prev] | [next] | [standalone]

#99322

On 14/12/16 04:18, DFS wrote:
> Trying to read in a file of date string (in this case 10) and add them
> to an array
>
> input
> 22-May-1819
> 01-Jan-1268
> 03-May-1417
> 01-Mar-1250
> 05-Oct-1566
> 21-Feb-1629
> 23-Jun-1478
> 18-Feb-1576
> 07-Jan-1099
> 21-Mar-1299
>
>
> =====================================================
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
>
> int main()
> {
>   int i=0,n=0;
>   char *datestr[20];  //segfaults unless 2*input lines
>   char line[12];

Melzzzzz and Mark have already pointed out that this isn't big enough to 
read a complete line of data (including the newline character), and 
increasing it to 13 will solve one of your problems. But they both 
missed some points:

>   FILE *fp = fopen("datestrings.txt","r");

What if the file fails to open? You should test fp against NULL. If the 
file can't be opened, you can't read from it, and the attempt is likely 
to crash your program.

>   while(fgets(line,sizeof(line),fp)!=NULL) {
>     datestr[i]=malloc(sizeof(line));

You should check that this succeeds before attempting to use the pointer:

datestr[i] = malloc(sizeof line);
if(datestr[i] != NULL)
{

and make an intelligent decision about what to do if the allocation 
fails. I can't help wondering why you are dynamically allocating the 
width of the string, the length of which you know, but not the number of 
items in the array, which is more prone to change.

>     strcpy(datestr[i],line);

Is a newline part of the data? Do you really want to store it?

>     i++;n++;
>   }
>   fclose(fp);
>   for (i=0;i<n;i++){
>    printf("%s\n", datestr[i]);

And now, let's look at what you're printing. What did you read into 
position 0 of the array? Answer: you tried to read "22-May-1819\n" but 
fgets wouldn't put the newline into the array, saving it for the next 
element. Now, you print a newline yourself --- %s\n rather than %s --- 
so we would expect to see in your output the date ("22-May-1819") plus a 
newline ("\n"). Now we come onto element 1 of the array, which just 
contains a newline on its own ("\n"). Now, you print the data, whatever 
it might be, plus a newline. Since the data is a newline too, you get 
two newlines: "\n\n".

So your output so far is: "22-May-1819\n\n\n", and we'd therefore expect 
to see your dates separated by *two* blank lines:

DATE 0 + first newline
second newline
third newline
DATE 1 + first newline
second newline
third newline

etc.

So let's take a look at the data you posted:

>   }
> }
> =====================================================
>
> output
> 22-May-1819
>
>
> 01-Jan-1268
>
>
> 03-May-1417
>
>
> 01-Mar-1250

Prediction confirmed.

Solution:

(a) make your line buffer a touch bigger, so that there's room for the 
newline when each line is read; AND
(b) EITHER get rid of that newline before you process your data, OR 
don't print an extra newline --- use %s rather than %s\n --- when you 
print the data. (Choose one solution or the other. If you choose both, 
all your data will appear on one line!)

-- 
Richard Heathfield
Email: rjh at cpax dot org dot uk
"Usenet is a strange place" - dmr 29 July 1999
Sig line 4 vacant - apply within

[toc] | [prev] | [next] | [standalone]

#99363

On 12/14/2016 05:38 AM, Richard Heathfield wrote:
> On 14/12/16 04:18, DFS wrote:
>> Trying to read in a file of date string (in this case 10) and add them
>> to an array
>>
>> input
>> 22-May-1819
>> 01-Jan-1268
>> 03-May-1417
>> 01-Mar-1250
>> 05-Oct-1566
>> 21-Feb-1629
>> 23-Jun-1478
>> 18-Feb-1576
>> 07-Jan-1099
>> 21-Mar-1299
>>
>>
>> =====================================================
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <string.h>
>>
>> int main()
>> {
>>   int i=0,n=0;
>>   char *datestr[20];  //segfaults unless 2*input lines
>>   char line[12];
>
> Melzzzzz and Mark have already pointed out that this isn't big enough to
> read a complete line of data (including the newline character), and
> increasing it to 13 will solve one of your problems. But they both
> missed some points:
>
>>   FILE *fp = fopen("datestrings.txt","r");
>
> What if the file fails to open? You should test fp against NULL. If the
> file can't be opened, you can't read from it, and the attempt is likely
> to crash your program.
>
>>   while(fgets(line,sizeof(line),fp)!=NULL) {
>>     datestr[i]=malloc(sizeof(line));
>
> You should check that this succeeds before attempting to use the pointer:
>
> datestr[i] = malloc(sizeof line);
> if(datestr[i] != NULL)
> {
>
> and make an intelligent decision about what to do if the allocation
> fails. I can't help wondering why you are dynamically allocating the
> width of the string, the length of which you know, but not the number of
> items in the array, which is more prone to change.
>
>>     strcpy(datestr[i],line);
>
> Is a newline part of the data? Do you really want to store it?

No, but I don't know how NOT to store it.



>>     i++;n++;
>>   }
>>   fclose(fp);
>>   for (i=0;i<n;i++){
>>    printf("%s\n", datestr[i]);
>
> And now, let's look at what you're printing. What did you read into
> position 0 of the array? Answer: you tried to read "22-May-1819\n" but
> fgets wouldn't put the newline into the array, saving it for the next
> element. Now, you print a newline yourself --- %s\n rather than %s ---
> so we would expect to see in your output the date ("22-May-1819") plus a
> newline ("\n"). Now we come onto element 1 of the array, which just
> contains a newline on its own ("\n"). Now, you print the data, whatever
> it might be, plus a newline. Since the data is a newline too, you get
> two newlines: "\n\n".
>
> So your output so far is: "22-May-1819\n\n\n", and we'd therefore expect
> to see your dates separated by *two* blank lines:
>
> DATE 0 + first newline
> second newline
> third newline
> DATE 1 + first newline
> second newline
> third newline
>
> etc.
>
> So let's take a look at the data you posted:
>
>>   }
>> }
>> =====================================================
>>
>> output
>> 22-May-1819
>>
>>
>> 01-Jan-1268
>>
>>
>> 03-May-1417
>>
>>
>> 01-Mar-1250
>
> Prediction confirmed.
>
> Solution:
>
> (a) make your line buffer a touch bigger, so that there's room for the
> newline when each line is read; AND
> (b) EITHER get rid of that newline before you process your data, OR
> don't print an extra newline --- use %s rather than %s\n --- when you
> print the data. (Choose one solution or the other. If you choose both,
> all your data will appear on one line!)


Thanks.  That worked.

I updated the program to count the lines in the input file, then set the 
array size to the linecount.  Plus added a little error checking.

The new code reads and printfs up to 1 million lines just fine, but 
bombs somewhere between 1 million and 1.1 million.


$ ./builddates 1100000 datestrings.txt

$ ./readdates datestringsXXX.txt
Failed to open file

$ ./readdates datestrings.txt
1100000 lines in file 'datestrings.txt'
Segmentation fault (core dumped)


Any idea why it crashes above 1 million?


readdates.c
================================================================
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char *argv[])
{

	int i=0,n=0,linewidth=13,linecount=0;
   char line[linewidth];
	char txtfile[strlen(argv[1])];
	FILE *fp; char c;

	//open file
	strcpy(txtfile,argv[1]);
	fp=fopen(txtfile,"r");
	if(fp==NULL){
	  printf("Failed to open file\n");
	  return -1;
	}


	//count lines in file
	c=getc(fp);
	while (c!=EOF){
	  if(c=='\n'){linecount+=1;}
	  c=getc(fp);
	}
	fclose(fp);
	printf("%d lines in file '%s'\n",linecount,txtfile);


	//copy data into array
	char *datestr[linecount];
	fp = fopen(txtfile,"r");
	while(fgets(line,linewidth,fp)!=NULL) {
	 datestr[i]=malloc(linewidth);
	 strcpy(datestr[i],line);
	 i++;n++;
	}
	fclose(fp);

	//print array
   for (i=0;i<n;i++){printf("%s", datestr[i]);}
}
================================================================

[toc] | [prev] | [next] | [standalone]

#99364

On 14/12/16 17:53, DFS wrote:
> On 12/14/2016 05:38 AM, Richard Heathfield wrote:
>> On 14/12/16 04:18, DFS wrote:
>>> Trying to read in a file of date string (in this case 10) and add them
>>> to an array
>>>
>>> input
>>> 22-May-1819
>>> 01-Jan-1268
>>> 03-May-1417
>>> 01-Mar-1250
>>> 05-Oct-1566
>>> 21-Feb-1629
>>> 23-Jun-1478
>>> 18-Feb-1576
>>> 07-Jan-1099
>>> 21-Mar-1299
>>>
>>>
>>> =====================================================
>>> #include <stdio.h>
>>> #include <stdlib.h>
>>> #include <string.h>
>>>
>>> int main()
>>> {
>>>   int i=0,n=0;
>>>   char *datestr[20];  //segfaults unless 2*input lines
>>>   char line[12];
>>
>> Melzzzzz and Mark have already pointed out that this isn't big enough to
>> read a complete line of data (including the newline character), and
>> increasing it to 13 will solve one of your problems. But they both
>> missed some points:
>>
>>>   FILE *fp = fopen("datestrings.txt","r");
>>
>> What if the file fails to open? You should test fp against NULL. If the
>> file can't be opened, you can't read from it, and the attempt is likely
>> to crash your program.
>>
>>>   while(fgets(line,sizeof(line),fp)!=NULL) {
>>>     datestr[i]=malloc(sizeof(line));
>>
>> You should check that this succeeds before attempting to use the pointer:
>>
>> datestr[i] = malloc(sizeof line);
>> if(datestr[i] != NULL)
>> {
>>
>> and make an intelligent decision about what to do if the allocation
>> fails. I can't help wondering why you are dynamically allocating the
>> width of the string, the length of which you know, but not the number of
>> items in the array, which is more prone to change.
>>
>>>     strcpy(datestr[i],line);
>>
>> Is a newline part of the data? Do you really want to store it?
>
> No, but I don't know how NOT to store it.

#include <string.h>

/* strip a newline from a string:
    return 0 if this was achieved,
    otherwise 1.
  */

int nlstrip(char *s)
{
   char *p = NULL;
   if(s != NULL)
   {
     p = strchr(s, '\n');
   }
   if(p != NULL)
   {
     *p = '\0';
   }
   return p == NULL;
}

Usage: nlstrip(line);

After the fgets but before the malloc.

> I updated the program to count the lines in the input file,

That approach may strike you as being a little inelegant, but for now it 
will do. Later, you can learn how to resize a dynamically allocated 
array as you go along. (This is a common technique.)

> then set the
> array size to the linecount.  Plus added a little error checking.
>
> The new code reads and printfs up to 1 million lines just fine, but
> bombs somewhere between 1 million and 1.1 million.
>
>
> $ ./builddates 1100000 datestrings.txt
>
> $ ./readdates datestringsXXX.txt
> Failed to open file
>
> $ ./readdates datestrings.txt
> 1100000 lines in file 'datestrings.txt'
> Segmentation fault (core dumped)
>
>
> Any idea why it crashes above 1 million?

Do you remember when I suggested checking the return value of malloc?

I again suggest that you check the return value of malloc.

The malloc function returns NULL when it fails to allocate memory. Since 
you're not checking for this, if ever it /does/ return NULL you will be 
scribbling all over memory you don't own, hence the segfault.

-- 
Richard Heathfield
Email: rjh at cpax dot org dot uk
"Usenet is a strange place" - dmr 29 July 1999
Sig line 4 vacant - apply within

[toc] | [prev] | [next] | [standalone]

#99387

Richard Heathfield <rjh@cpax.org.uk> writes:
> On 14/12/16 17:53, DFS wrote:
[...]
>> Any idea why it crashes above 1 million?
>
> Do you remember when I suggested checking the return value of malloc?
>
> I again suggest that you check the return value of malloc.
>
> The malloc function returns NULL when it fails to allocate memory. Since 
> you're not checking for this, if ever it /does/ return NULL you will be 
> scribbling all over memory you don't own, hence the segfault.

On the other hand, on some systems malloc() generally doesn't report
failure; rather if you allocate too much memory, a crash occurs when you
try to access it.  (This is arguably non-conforming.)

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#99392

On 14/12/16 19:57, Keith Thompson wrote:
> Richard Heathfield <rjh@cpax.org.uk> writes:
>> On 14/12/16 17:53, DFS wrote:
> [...]
>>> Any idea why it crashes above 1 million?
>>
>> Do you remember when I suggested checking the return value of malloc?
>>
>> I again suggest that you check the return value of malloc.
>>
>> The malloc function returns NULL when it fails to allocate memory. Since
>> you're not checking for this, if ever it /does/ return NULL you will be
>> scribbling all over memory you don't own, hence the segfault.
>
> On the other hand, on some systems malloc() generally doesn't report
> failure; rather if you allocate too much memory, a crash occurs when you
> try to access it.  (This is arguably non-conforming.)

Arguably? I'd have said it's absolutely definitely non-conforming. The 
malloc function is /required/ to report failure if the memory can't be 
allocated.

7.22.3(1): The order and contiguity of storage allocated by successive 
calls to the aligned_alloc, calloc, malloc, and realloc functions is 
unspecified. The pointer returned if the allocation succeeds is suitably 
aligned so that it may be assigned to a pointer to any type of object 
with a fundamental alignment requirement and then used to access such an 
object or an array of such objects in the space allocated (until the 
space is explicitly deallocated). The lifetime of an allocated object 
extends from the allocation until the deallocation. Each such allocation 
shall yield a pointer to an object disjoint from any other object. The 
pointer returned points to the start (lowest byte address) of the 
allocated space. If the space cannot be allocated, a null pointer is 
returned.

Note the last sentence: if the space can't be allocated, malloc has to 
return a null pointer. If it returns a non-null pointer, this can only 
mean that the allocation succeeded. So if malloc returns a non-null 
pointer when the allocation in fact failed, the implementation fails to 
conform to 7.22.3(1).

What intrigues me, though, is why you think this is arguable. I suspect 
that you probably share my take on this. If you were to play Devil's 
Advocate, though, what arguments would you muster for such behaviour 
being conforming?

-- 
Richard Heathfield
Email: rjh at cpax dot org dot uk
"Usenet is a strange place" - dmr 29 July 1999
Sig line 4 vacant - apply within

[toc] | [prev] | [next] | [standalone]

#99402

On 12/14/2016 03:05 PM, Richard Heathfield wrote:
> On 14/12/16 19:57, Keith Thompson wrote:
...
>> On the other hand, on some systems malloc() generally doesn't report
>> failure; rather if you allocate too much memory, a crash occurs when you
>> try to access it.  (This is arguably non-conforming.)
>
> Arguably? I'd have said it's absolutely definitely non-conforming. The
> malloc function is /required/ to report failure if the memory can't be
> allocated.
...
> What intrigues me, though, is why you think this is arguable. I suspect
> that you probably share my take on this. If you were to play Devil's
> Advocate, though, what arguments would you muster for such behaviour
> being conforming?

For myself, I would call it "arguable" for no better reason than the 
empirical fact that I've seen people argue about the issue. The fact 
that I think the other side's arguments don't hold water doesn't change 
the fact that they are, in fact, willing to argue about them.

I don't entirely discount the quality of the arguments - for instance, 
Jerry Stuckle routinely argues truly ridiculous points without ever 
bothering to present any stronger argument than "you're an idiot for 
disagreeing with me". If such are the only "arguments" I've seen on an 
issue, that's not enough for me to label it as "arguable". Unless you at 
least attempt to address the points made by the other side, it's just 
loud angry posturing, not an actual argument.

[toc] | [prev] | [next] | [standalone]

#99409

Richard Heathfield <rjh@cpax.org.uk> writes:
> On 14/12/16 19:57, Keith Thompson wrote:
>> Richard Heathfield <rjh@cpax.org.uk> writes:
>>> On 14/12/16 17:53, DFS wrote:
>> [...]
>>>> Any idea why it crashes above 1 million?
>>>
>>> Do you remember when I suggested checking the return value of malloc?
>>>
>>> I again suggest that you check the return value of malloc.
>>>
>>> The malloc function returns NULL when it fails to allocate memory. Since
>>> you're not checking for this, if ever it /does/ return NULL you will be
>>> scribbling all over memory you don't own, hence the segfault.
>>
>> On the other hand, on some systems malloc() generally doesn't report
>> failure; rather if you allocate too much memory, a crash occurs when you
>> try to access it.  (This is arguably non-conforming.)
>
> Arguably? I'd have said it's absolutely definitely non-conforming. The 
> malloc function is /required/ to report failure if the memory can't be 
> allocated.
[snip]
> What intrigues me, though, is why you think this is arguable. I suspect 
> that you probably share my take on this. If you were to play Devil's 
> Advocate, though, what arguments would you muster for such behaviour 
> being conforming?

I agree that it's non-conforming.  It's also much less useful than
having malloc() properly report failures.

I've seen a counterargument (which I can't necessarily summarize
accurately, but I'll try) that malloc() succeeded in allocating a
chunk of virtual memory.  The later failure to use that virtual
memory is a capacity failure, and the standard does not specify
"the size or complexity of a program and its data that will exceed
the capacity of any specific data-processing system or the capacity
of a particular processor".

Furthermore, it's possible that this:

    ptr = malloc(big_value);
    if (ptr != NULL) memset(ptr, 42, big_value);

could fail on the memset call, but this:

    ptr = malloc(big_value);
    do_some_other_stuff();
    if (ptr != NULL) memset(ptr, 42, big_value);

might succeed.  do_some_other_stuff() might free some memory,
or it might even be something that happens in other processes,
depending on whether the shortage applies to the current process
or the entire system.

I suppose I should mention the background.  This can happen,
as I understand it, on Linux-based systems.  If the attempt to
access the "allocated" memory fails, it invokes something called
the "OOM killer" (OOM is Out Of Memory), which will kill *some*
chosen process(es), not necessarily the one that caused the error.

It's entirely possible that I'm not describing this correctly, and
that there are valid reasons for this behavior.  Again, be aware
that I'm summarizing arguments that I don't necessarily agree with
or know very well, so take all this with a large grain of salt.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#99415

On Wednesday, December 14, 2016 at 2:29:02 PM UTC-6, Keith Thompson wrote:
> I've seen a counterargument (which I can't necessarily summarize
> accurately, but I'll try) that malloc() succeeded in allocating a
> chunk of virtual memory.  The later failure to use that virtual
> memory is a capacity failure, and the standard does not specify
> "the size or complexity of a program and its data that will exceed
> the capacity of any specific data-processing system or the capacity
> of a particular processor".

Nothing in the C Standard specifies how much stack a system must have, how
much stack might be used by various concepts, how much stack malloc() might
require, or what might happen if malloc() is invoked when insufficient stack
space is available.  Thus, there is no guarantee that any particular call to
malloc() will always either yield a usable pointer or NULL.  Any call could
run out of stack and do anything whatsoever, or behave as though it runs out
of stack and might do anything whatsoever.  A decent-quality implementation
will support malloc() calls from useful contexts, but the Standard does not
require any particular level of implementation quality.

Further, if an underlying platform lacks any safe means to request a block
of memory or indicate that it's available, there will be no way that a C
implementation for that platform will be able to expose such a feature to
the programmer.  I would suggest that it would be more useful to say that
such platforms can be used to run C programs that can survive without that
guarantee than to say that such a platform can't host a C implementation.
What would be more useful yet would be to have predefined macros that would
promise one of the following:

   1. A malloc() call which is not made from an excessively-deep nested
      context will either return NULL or return a pointer to a block that
      can be used safely.

   2. A malloc() call will always either succeed or invoke an implementation-
      defined trap whose default behavior is to force Abnormal Program
      Termination (referred to hereafter as an IDT/APT).

   3. A malloc() call will always either invoke an IDT/APT or yield a pointer
      for which accesses will either succeed or invoke an IDT/APT.

   4. A malloc() call may do any of the above, chosen in arbitrary
      fashion.

   5. A malloc() call will not prevent a program from compiling, but may
      invoke Undefined Behavior in case of memory depletion.

If an implementation promises #1, programs for it may be able to guarantee
clean behavior even in out-of-memory conditions.  If an implementation
promises #2 or #3, programs for it may omit null checks on pointers
returned from malloc().  If an implementation can't promise #4, any use
of malloc() may pose unavoidable risks, but for some applications that might
be acceptable.

[toc] | [prev] | [next] | [standalone]

#99418

Keith Thompson <kst-u@mib.org> writes:

> I suppose I should mention the background.  This can happen,
> as I understand it, on Linux-based systems.  If the attempt to
> access the "allocated" memory fails, it invokes something called
> the "OOM killer" (OOM is Out Of Memory), which will kill *some*
> chosen process(es), not necessarily the one that caused the error.

It is configurable (vm.overcommit_memory and some other parameters).  The
default potentially depend on the distribution (here I think the default
was heuristic before I changed it).  My understanding is that with "never",
there are common situations which require far more memory than needed (ISTR
for instance forking will require the possibility to really duplicate the
process although it will often be followed by a fork) but I don't remember
having had issues with "never"; note that I've setup a swap and rarely I
push the system so that swapping is needed.

Excepted the "temporary need for fork", the other major use case (excepted
lazy memory handling) seems to be applications which are playing tricks
with the virtual memory such as using a large sparse array.

ISTR that AIX also has that feature, 

Yours,

-- 
Jean-Marc

[toc] | [prev] | [next] | [standalone]

#99426

Jean-Marc Bourguet <jm@bourguet.org> wrote:
> Keith Thompson <kst-u@mib.org> writes:
> 
>> I suppose I should mention the background.  This can happen,
>> as I understand it, on Linux-based systems.  If the attempt to
>> access the "allocated" memory fails, it invokes something called
>> the "OOM killer" (OOM is Out Of Memory), which will kill *some*
>> chosen process(es), not necessarily the one that caused the error.
<snip>
>
> Excepted the "temporary need for fork", the other major use case (excepted
> lazy memory handling) seems to be applications which are playing tricks
> with the virtual memory such as using a large sparse array.
> 

That's the real reason, not the fork issue. The root of the problem,
supposedly, is applications that preallocate huge amounts of memory as a
performance optimization. For example, a database. A more recent example
would be some virtual machine memory managers that preallocate a large
amount of memory.

Often those applications will only ever touch a small amount of the memory.
And of course forked processes in particular are unlikely to touch any of
it. The issue with fork simply highlights the problem. Failing allocation or
fork would in most cases just be a needless inconvenience.

I don't find the argument very persuasive. Maybe the argument made more
sense 20 years ago, especially when Linux was trying to run big-iron
software on tiny desktops. But today preallocating memory like that is
frowned upon, and in any event there are much richer virtual memory
interfaces available to applications (e.g. mmap) that allow them to tell the
kernel if and how such memory should be inherited. Modern databases and
virtual machines know how to use these interfaces properly.

Virtual memory managers are much more sophisticated wrt sharing mappings for
regular files, including program code. In modern kernels the disk buffer
cache is shared with the virtual memory page cache, so memory-mapped files
can be serviced directly from disk, and kernels don't need to count those
pages against a process.

Applications can even effectively create their own swap space by creating,
deleting, and mapping large regular files. If the application truly never
touches the memory there's unlikely to be much if any disk activity. You get
all the benefits of overcommit, without the negatives.

And vfork is still a thing, not to mention posix_spawn.

[toc] | [prev] | [next] | [standalone]

#99423

Richard Heathfield <rjh@cpax.org.uk> writes:

> On 14/12/16 19:57, Keith Thompson wrote:
>> Richard Heathfield <rjh@cpax.org.uk> writes:
>>> On 14/12/16 17:53, DFS wrote:
>> [...]
>>>> Any idea why it crashes above 1 million?
>>>
>>> Do you remember when I suggested checking the return value of malloc?
>>>
>>> I again suggest that you check the return value of malloc.
>>>
>>> The malloc function returns NULL when it fails to allocate memory. Since
>>> you're not checking for this, if ever it /does/ return NULL you will be
>>> scribbling all over memory you don't own, hence the segfault.
>>
>> On the other hand, on some systems malloc() generally doesn't report
>> failure; rather if you allocate too much memory, a crash occurs when you
>> try to access it.  (This is arguably non-conforming.)
>
> Arguably? I'd have said it's absolutely definitely non-conforming. The
> malloc function is /required/ to report failure if the memory can't be
> allocated.
>
> 7.22.3(1): The order and contiguity of storage allocated by successive
> calls to the aligned_alloc, calloc, malloc, and realloc functions is
> unspecified. The pointer returned if the allocation succeeds is
> suitably aligned so that it may be assigned to a pointer to any type
> of object with a fundamental alignment requirement and then used to
> access such an object or an array of such objects in the space
> allocated (until the space is explicitly deallocated). The lifetime of
> an allocated object extends from the allocation until the
> deallocation. Each such allocation shall yield a pointer to an object
> disjoint from any other object. The pointer returned points to the
> start (lowest byte address) of the allocated space. If the space
> cannot be allocated, a null pointer is returned.
>
> Note the last sentence: if the space can't be allocated, malloc has to
> return a null pointer. If it returns a non-null pointer, this can only
> mean that the allocation succeeded. So if malloc returns a non-null
> pointer when the allocation in fact failed, the implementation fails
> to conform to 7.22.3(1).
>
> What intrigues me, though, is why you think this is arguable. I
> suspect that you probably share my take on this. If you were to play
> Devil's Advocate, though, what arguments would you muster for such
> behaviour being conforming?

The last sentence actually isn't why I regard it as non-comforming.
Earlier, it says "... and then used to access such an object or an array
of such objects in the space allocated...".  The space has been
successfully allocated (in the virtual memory of the process", but Linux
provides no guarantee that it can later be accessed.

You can turn off overcommit, however.

[toc] | [prev] | [next] | [standalone]

#99425

On 14/12/16 22:41, Joe Pfeiffer wrote:
<snip>
>
> You can turn off overcommit, however.

Have you ever tried?

I did, and practically every application I tried to use would fall over 
within a few minutes (or, sometimes, within a few seconds).

I do worry sometimes. So many people think that features are more 
important than robustness. The prevailing attitude seems to be that, if 
there isn't enough memory to do /everything/, we'll damn well do 
nothing. Not enough memory for a new tab in the browser, say? What 
should we do? Put up a dialog saying "sorry, can't do that, but we'll 
leave the other tabs open so you can keep on doing whatever you were 
doing"? No, that sounds too sensible. Let's just crash and burn, and to 
hell with the user.

Oh well. C'est la vie.

-- 
Richard Heathfield
Email: rjh at cpax dot org dot uk
"Usenet is a strange place" - dmr 29 July 1999
Sig line 4 vacant - apply within

[toc] | [prev] | [next] | [standalone]

#99428

Richard Heathfield <rjh@cpax.org.uk> writes:

> On 14/12/16 22:41, Joe Pfeiffer wrote:
> <snip>
>>
>> You can turn off overcommit, however.
>
> Have you ever tried?

No.

> I did, and practically every application I tried to use would fall
> over within a few minutes (or, sometimes, within a few seconds).

That's weird.  How much swap did you have?  The only time I've seen
behavior like that was when a system had swapped turned off (I wasn't
the sysadmin!).

> I do worry sometimes. So many people think that features are more
> important than robustness. The prevailing attitude seems to be that,
> if there isn't enough memory to do /everything/, we'll damn well do
> nothing. Not enough memory for a new tab in the browser, say? What
> should we do? Put up a dialog saying "sorry, can't do that, but we'll
> leave the other tabs open so you can keep on doing whatever you were
> doing"? No, that sounds too sensible. Let's just crash and burn, and
> to hell with the user.
>
> Oh well. C'est la vie.

Well, they're expecting overcommit.  It's pretty close to true
(especially on a 64 bit system) with overcommit that the only time
malloc() will fail is if something has corrupted the memory pool; in
that case, you might as well crash now since you're going to very soon
(or worse).

[toc] | [prev] | [next] | [standalone]

#99429

On 15/12/16 00:33, Joe Pfeiffer wrote:
> Richard Heathfield <rjh@cpax.org.uk> writes:
>
>> On 14/12/16 22:41, Joe Pfeiffer wrote:
>> <snip>
>>>
>>> You can turn off overcommit, however.
>>
>> Have you ever tried?
>
> No.
>
>> I did, and practically every application I tried to use would fall
>> over within a few minutes (or, sometimes, within a few seconds).
>
> That's weird.  How much swap did you have?

Several tons. Truckloads.

> Well, they're expecting overcommit.

Well, they shouldn't.

> It's pretty close to true
> (especially on a 64 bit system) with overcommit that the only time
> malloc() will fail is if something has corrupted the memory pool; in
> that case, you might as well crash now since you're going to very soon
> (or worse).

Or you might as well not bother testing, which is what I suspect happens 
most of the time. And that's why Linux software is so hopeless. Not as 
hopeless as Windows software, obviously --- but still pretty hopeless. 
Look! Shiny!

-- 
Richard Heathfield
Email: rjh at cpax dot org dot uk
"Usenet is a strange place" - dmr 29 July 1999
Sig line 4 vacant - apply within

[toc] | [prev] | [next] | [standalone]

#99431

On Wednesday, December 14, 2016 at 6:42:16 PM UTC-6, Richard Heathfield wrote:
> Or you might as well not bother testing, which is what I suspect happens 
> most of the time. And that's why Linux software is so hopeless. Not as 
> hopeless as Windows software, obviously --- but still pretty hopeless. 
> Look! Shiny!

There are many kinds of application that won't be able to do anything useful
unless all of their malloc() calls succeed.  In such applications, having
allocation failures force an abnormal program termination may be more helpful
than having them return null (requiring that every malloc() call check its
result and--if null--force an abnormal program termination).

Abnormal program termination is hardly elegant, but if a program isn't going
to be able to behave usefully, the value of avoiding an APE may be limited.

[toc] | [prev] | [next] | [standalone]

#99454

On 15/12/16 01:05, supercat@casperkitty.com wrote:
> On Wednesday, December 14, 2016 at 6:42:16 PM UTC-6, Richard Heathfield wrote:
>> Or you might as well not bother testing, which is what I suspect happens
>> most of the time. And that's why Linux software is so hopeless. Not as
>> hopeless as Windows software, obviously --- but still pretty hopeless.
>> Look! Shiny!
>
> There are many kinds of application that won't be able to do anything useful
> unless all of their malloc() calls succeed.

I think that depends on what you mean by "application". If you just mean 
"program", I agree. But nowadays, it's probably fair to say that 
"application" means "event-driven, interactive program" - word 
processors, spreadsheets, browsers, desktop publishing programs, music 
editors, arts packages, photographic editors, and all the rest of it. 
And these programs certainly can continue to be useful even if some of 
their malloc calls fail. If a photo editor, say, can't open a photograph 
because it's too darn huge, I'd appreciate being told that in a dialog 
window that I can dismiss to carry on working on some other photos, 
rather than being told it by the crashing of the application before I 
have so much as a chance to save my work.

> In such applications, having
> allocation failures force an abnormal program termination may be more helpful
> than having them return null (requiring that every malloc() call check its
> result and--if null--force an abnormal program termination).

Useful and convenient to the programmer, maybe, but not even remotely 
useful to the user, and it is the user to whom the program is /supposed/ 
to be useful. As for checking the result of malloc and then deliberately 
aborting if it's NULL, well, that's precisely the kind of thinking that 
leads to the flaky applications we have to use every day because we lack 
the several hundred person-years of spare time it would take to go and 
fix all the source code.

> Abnormal program termination is hardly elegant, but if a program isn't going
> to be able to behave usefully, the value of avoiding an APE may be limited.

Abnormal program termination isn't just inelegant. It's very often a 
sign of lazy and thoughtless programming. It's fine for debugging and 
testing, but it has no place in production code.

-- 
Richard Heathfield
Email: rjh at cpax dot org dot uk
"Usenet is a strange place" - dmr 29 July 1999
Sig line 4 vacant - apply within

[toc] | [prev] | [next] | [standalone]

#99532

On Thursday, December 15, 2016 at 2:21:11 AM UTC-6, Richard Heathfield wrote:
> On 15/12/16 01:05, supercat wrote:
> > There are many kinds of application that won't be able to do anything useful
> > unless all of their malloc() calls succeed.
> 
> I think that depends on what you mean by "application". If you just mean 
> "program", I agree. But nowadays, it's probably fair to say that 
> "application" means "event-driven, interactive program" - word 
> processors, spreadsheets, browsers, desktop publishing programs, music 
> editors, arts packages, photographic editors, and all the rest of it. 
> And these programs certainly can continue to be useful even if some of 
> their malloc calls fail. 

Not all applications are interactive programs whose memory usage will vary
based upon user input.  There are many other kinds which either do not
operate interactively, or which pre-allocate the storage they're ever going
to need, and for those kinds of programs memory will either be adequate or
it won't.

>If a photo editor, say, can't open a photograph 
> because it's too darn huge, I'd appreciate being told that in a dialog 
> window that I can dismiss to carry on working on some other photos, 
> rather than being told it by the crashing of the application before I 
> have so much as a chance to save my work.

In order for that to work robustly, the program would have to have different
means of requesting allocations for things which are allowed to fail, and
for things which aren't.  If all of the allocations made by a program have
succeeded up until the user selects "Save as...", but there isn't enough
memory left to display the dialog box, what should happen?

What's needed is not to say that the program needs to prepare for every
allocation request to fail, but rather that a program needs to have a means
of ensuring that certain requests *won't* fail, and that requests which
could fail safely will only succeed in cases that might cause other, more
important, requests to fail.

> Useful and convenient to the programmer, maybe, but not even remotely 
> useful to the user, and it is the user to whom the program is /supposed/ 
> to be useful. As for checking the result of malloc and then deliberately 
> aborting if it's NULL, well, that's precisely the kind of thinking that 
> leads to the flaky applications we have to use every day because we lack 
> the several hundred person-years of spare time it would take to go and 
> fix all the source code.

How should the code for a "Save as..." dialog box behave if it runs out of
memory after creating a few of the graphical resources it requires?  Is it
more helpful to check each individual allocation for success, or instead
focusing on ways of ensuring that memory doesn't get committed to "optional"
purposes unless there's certain to be enough to satisfy all necessary ones?

> > Abnormal program termination is hardly elegant, but if a program isn't going
> > to be able to behave usefully, the value of avoiding an APE may be limited.
> 
> Abnormal program termination isn't just inelegant. It's very often a 
> sign of lazy and thoughtless programming. It's fine for debugging and 
> testing, but it has no place in production code.

That depends upon the application.  There are certainly some kinds that can
and must continue functioning usefully even after an allocation failure, but
if an application knows in advance of receiving input how much memory it is
going to need, and can't do anything useful without it, having the program
die with a system-generated "Out of memory error" isn't much less useful than
having a program-generated "Sorry, but I can't allocate enough memory."

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.c

csiph-web