Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #11885 > unrolled thread

Analogue of ReadLine() in C

Back to article view | Back to comp.lang.c

Contents

  Analogue of ReadLine() in C Максим Фомин <maxim@maxim-fomin.ru> - 2011-10-19 12:26 -0700
    Re: Analogue of ReadLine() in C China Blue Corn Chips <chine.bleu@yahoo.com> - 2011-10-19 13:05 -0700
      Re: Analogue of ReadLine() in C China Blue Corn Chips <chine.bleu@yahoo.com> - 2011-10-19 20:50 -0700
        Re: Analogue of ReadLine() in C Seebs <usenet-nospam@seebs.net> - 2011-10-20 07:42 +0000
          Re: Analogue of ReadLine() in C China Blue Corn Chips <chine.bleu@yahoo.com> - 2011-10-20 02:51 -0700
            Re: Analogue of ReadLine() in C gazelle@shell.xmission.com (Kenny McCormack) - 2011-10-20 12:52 +0000
            Re: Analogue of ReadLine() in C "BartC" <bc@freeuk.com> - 2011-10-20 14:31 +0100
              Re: Analogue of ReadLine() in C Malcolm McLean <malcolm.mclean5@btinternet.com> - 2011-10-20 08:05 -0700
                Re: Analogue of ReadLine() in C gazelle@shell.xmission.com (Kenny McCormack) - 2011-10-20 15:15 +0000
                Re: Analogue of ReadLine() in C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-10-20 16:50 +0100
                Re: Analogue of ReadLine() in C "BartC" <bc@freeuk.com> - 2011-10-20 23:23 +0100
                  Re: Analogue of ReadLine() in C Keith Thompson <kst-u@mib.org> - 2011-10-20 15:34 -0700
                    Re: Analogue of ReadLine() in C Ian Collins <ian-news@hotmail.com> - 2011-10-21 14:10 +1300
                    Re: Analogue of ReadLine() in C "BartC" <bc@freeuk.com> - 2011-10-21 10:30 +0100
                      Re: Analogue of ReadLine() in C Malcolm McLean <malcolm.mclean5@btinternet.com> - 2011-10-21 05:59 -0700
                      Re: Analogue of ReadLine() in C Keith Thompson <kst-u@mib.org> - 2011-10-21 10:58 -0700
                        Re: Analogue of ReadLine() in C "BartC" <bc@freeuk.com> - 2011-10-22 20:00 +0100
                          Re: Analogue of ReadLine() in C Ian Collins <ian-news@hotmail.com> - 2011-10-23 08:09 +1300
                          Re: Analogue of ReadLine() in C Malcolm McLean <malcolm.mclean5@btinternet.com> - 2011-10-22 23:55 -0700
              Re: Analogue of ReadLine() in C Ian Collins <ian-news@hotmail.com> - 2011-10-21 08:34 +1300
            Re: Analogue of ReadLine() in C Seebs <usenet-nospam@seebs.net> - 2011-10-20 17:57 +0000
            Re: Analogue of ReadLine() in C Markus Wichmann <nullplan@gmx.net> - 2011-10-23 23:03 +0200
              Re: Analogue of ReadLine() in C Acid Washed China Blue Jeans <chine.bleu@yahoo.com> - 2011-10-24 00:31 -0700
              Re: Analogue of ReadLine() in C Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-24 08:10 -0400
    Re: Analogue of ReadLine() in C 88888 Dihedral <dihedral88888@googlemail.com> - 2011-10-19 14:04 -0700
      Re: Analogue of ReadLine() in C Kaz Kylheku <kaz@kylheku.com> - 2011-10-19 21:52 +0000
    Re: Analogue of ReadLine() in C Ike Naar <ike@iceland.freeshell.org> - 2011-10-19 22:34 +0000
    Re: Analogue of ReadLine() in C pete <pfiland@mindspring.com> - 2011-10-19 19:02 -0400
      Re: Analogue of ReadLine() in C pete <pfiland@mindspring.com> - 2011-10-20 00:36 -0400
    Re: Analogue of ReadLine() in C Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-10-20 00:11 -0700
      Re: Analogue of ReadLine() in C Максим Фомин <maxim@maxim-fomin.ru> - 2011-10-20 01:59 -0700
    Re: Analogue of ReadLine() in C James Kuyper <jameskuyper@verizon.net> - 2011-10-20 06:57 -0400
    Re: Analogue of ReadLine() in C Markus Wichmann <nullplan@gmx.net> - 2011-10-20 18:40 +0200
      Re: Analogue of ReadLine() in C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-10-20 18:20 +0100
        Re: Analogue of ReadLine() in C Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-10-21 08:58 +0000
          Re: Analogue of ReadLine() in C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-10-21 19:12 +0100
          Re: Analogue of ReadLine() in C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-10-21 19:35 +0100
            Re: Analogue of ReadLine() in C "io_x" <a@b.c.invalid> - 2011-10-22 18:58 +0200
          Re: Analogue of ReadLine() in C David Thompson <dave.thompson2@verizon.net> - 2011-10-28 01:34 -0400
    Re: Analogue of ReadLine() in C Michael Angelo Ravera <maravera@prodigy.net> - 2011-10-20 13:32 -0700
      Re: Analogue of ReadLine() in C James Kuyper <jameskuyper@verizon.net> - 2011-10-20 16:42 -0400
      Re: Analogue of ReadLine() in C "Morris Keesan" <mkeesan@post.harvard.edu> - 2011-10-20 16:47 -0400
    Re: Analogue of ReadLine() in C Shao Miller <sha0.miller@gmail.com> - 2012-01-07 14:37 -0500

Page 1 of 3  [1] 2 3  Next page →

#11885 — Analogue of ReadLine() in C

I need function similar to ReadLine() in any other language.
The program below fails[1] when data more then LAG_SIZE is passed

#include <stdio.h>
#include <stdbool.h>
#include <string.h>
#include <stdlib.h>

char * read_line(void)
{
	bool read_more = true;
	enum { LAG_SIZE = 32 };
	char *return_buf = NULL;
	char *new_line_pos = NULL;
	char tmp_buf[LAG_SIZE] = {'\0'};
	size_t size = 0;
	while (read_more && fgets(tmp_buf, LAG_SIZE, stdin)){
		char *tmp_realloc;
		size += LAG_SIZE;
		tmp_realloc = realloc(return_buf, size);
		if (!tmp_realloc){
			free(return_buf);
			return_buf = NULL;
			break;
		}
		return_buf = tmp_realloc;
		memset(return_buf + size - LAG_SIZE, '\0', LAG_SIZE);
		new_line_pos = strchr(tmp_buf, '\n');
		if (new_line_pos){
			*new_line_pos = '\0';
			read_more = false;
		}
		strcat(return_buf, tmp_buf);
		memset(tmp_buf, '\0', LAG_SIZE);
	}
	return return_buf;
}

void loop(void)
{
	bool repeat = true;
	while (repeat){
		char *command;
		printf("Enter> ");
		command = read_line();
		if (command && !strcmp(command, "exit"))
			repeat = false;
		free(command);
	}
}

int main(void)
{
	loop();
	return 0;
}

[1]: *** glibc detected *** ./a.out: realloc(): invalid next size:
0x0804b008 ***

[toc] | [next] | [standalone]

#11887

In article <91b648bd-bf1e-4093-97a4-77b7300a94c5@c1g2000vbw.googlegroups.com>,
 埒峷彧挸弳恚 埇挭恚弳恧 <maxim@maxim-fomin.ru> wrote:

> I need function similar to ReadLine() in any other language.
> The program below fails[1] when data more then LAG_SIZE is passed

char *rdl(FILE *file) {
    char *p = 0; int m = 0, n = 0, c;
    while (c=fgetc(file), c!='\n' && c!=EOF) {
        if (n>=m) {m = 2*n+1; p = realloc(p, m);}
        p[n++] = c;
    }
    if (n>0 || c=='\n') {
        p = realloc(p, n+1); p[n] = 0;
    }
    return p;
}



> 
> #include <stdio.h>
> #include <stdbool.h>
> #include <string.h>
> #include <stdlib.h>
> 
> char * read_line(void)
> {
> 	bool read_more = true;
> 	enum { LAG_SIZE = 32 };
> 	char *return_buf = NULL;
> 	char *new_line_pos = NULL;
> 	char tmp_buf[LAG_SIZE] = {'\0'};
> 	size_t size = 0;
> 	while (read_more && fgets(tmp_buf, LAG_SIZE, stdin)){
> 		char *tmp_realloc;
> 		size += LAG_SIZE;
> 		tmp_realloc = realloc(return_buf, size);
> 		if (!tmp_realloc){
> 			free(return_buf);
> 			return_buf = NULL;
> 			break;
> 		}
> 		return_buf = tmp_realloc;
> 		memset(return_buf + size - LAG_SIZE, '\0', LAG_SIZE);
> 		new_line_pos = strchr(tmp_buf, '\n');
> 		if (new_line_pos){
> 			*new_line_pos = '\0';
> 			read_more = false;
> 		}
> 		strcat(return_buf, tmp_buf);
> 		memset(tmp_buf, '\0', LAG_SIZE);
> 	}
> 	return return_buf;
> }
> 
> void loop(void)
> {
> 	bool repeat = true;
> 	while (repeat){
> 		char *command;
> 		printf("Enter> ");
> 		command = read_line();
> 		if (command && !strcmp(command, "exit"))
> 			repeat = false;
> 		free(command);
> 	}
> }
> 
> int main(void)
> {
> 	loop();
> 	return 0;
> }
> 
> [1]: *** glibc detected *** ./a.out: realloc(): invalid next size:
> 0x0804b008 ***

-- 
Celle est une langue.     |   With the nutty taste of wild hickory nuts.
C'est francais           /|\              I'm whoever you want me to be.
et tres, tres sexuel.    \|/         Annoying Usenet one post at a time.
- China Azul              |            At least I can stay in character.

[toc] | [prev] | [next] | [standalone]

#11896

In article <realloc-20111020000027@ram.dialup.fu-berlin.de>,
 ram@zedat.fu-berlin.de (Stefan Ram) wrote:

> China Blue Corn Chips <chine.bleu@yahoo.com> writes:
> >char *p = 0; int m = 0, n = 0, c;
> >while (c=fgetc(file), c!='\n' && c!=EOF) {
> >    if (n>=m) {m = 2*n+1; p = realloc(p, m);}
> >    p[n++] = c;
> 
>   What if !realloc(p,m)?

Buy a bigger swap disk.

-- 
Celle est une langue.     |   With the nutty taste of wild hickory nuts.
C'est francais           /|\              I'm whoever you want me to be.
et tres, tres sexuel.    \|/         Annoying Usenet one post at a time.
- China Azul              |            At least I can stay in character.

[toc] | [prev] | [next] | [standalone]

#11902

On 2011-10-20, China Blue Corn Chips <chine.bleu@yahoo.com> wrote:
> In article <realloc-20111020000027@ram.dialup.fu-berlin.de>,
>  ram@zedat.fu-berlin.de (Stefan Ram) wrote:
>> China Blue Corn Chips <chine.bleu@yahoo.com> writes:
>> >char *p = 0; int m = 0, n = 0, c;
>> >while (c=fgetc(file), c!='\n' && c!=EOF) {
>> >    if (n>=m) {m = 2*n+1; p = realloc(p, m);}
>> >    p[n++] = c;

>>   What if !realloc(p,m)?

> Buy a bigger swap disk.

This is crappy advice, because it's completely unrelated.  There are
tons of systems which, for sane and well-considered reasons, will cap
allocations far below the size of physical memory or swap.  2*n+1 goes
pathological pretty fast.  Also, it's... ridiculous, frankly, to start
at 0.  That guarantees at least one realloc, and probably several for
even a tiny string.

Imagine that you're on a system which, quite reasonably, limits your
32-bit code to 3GB of memory.  You can have all 3GB, but you can't have
more.

Now imagine that you want to process a string which contains 1GB of
data.  Your algorithm will do ~30 reallocs, most of which will almost
certainly be copying all that data, and then fail.  Think it through;
what happens?  At any given time, your current length is a value of
the form 2^n - 1.  So what's it going to do for 2^n bytes of data,
say, 1GB exactly?

1.  It reallocs until it has exactly one byte less than 1GB of data.
2.  It then tries to allocate exactly one byte less than 2GB of data,
to copy that first GB into.

But that's two bytes less than 3GB of data it needs to be able to handle,
and I doubt your code and support libraries fit into 2 bytes.

Depending, I'd say either start with a "likely" length or start with
something around a page size, and reallocate up a little more conservatively.

-s
-- 
Copyright 2011, all wrongs reversed.  Peter Seebach / usenet-nospam@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.

[toc] | [prev] | [next] | [standalone]

#11904

In article <slrnj9vjr9.2vs.usenet-nospam@guild.seebs.net>,
 Seebs <usenet-nospam@seebs.net> wrote:

> On 2011-10-20, China Blue Corn Chips <chine.bleu@yahoo.com> wrote:
> > In article <realloc-20111020000027@ram.dialup.fu-berlin.de>,
> >  ram@zedat.fu-berlin.de (Stefan Ram) wrote:
> >> China Blue Corn Chips <chine.bleu@yahoo.com> writes:
> >> >char *p = 0; int m = 0, n = 0, c;
> >> >while (c=fgetc(file), c!='\n' && c!=EOF) {
> >> >    if (n>=m) {m = 2*n+1; p = realloc(p, m);}
> >> >    p[n++] = c;
> 
> >>   What if !realloc(p,m)?
> 
> > Buy a bigger swap disk.
> 
> This is crappy advice, because it's completely unrelated.  There are

The maximum line size is the file size. If the VM that can map in the entire 
file, it can allocate space for one line. If you want to run on smaller 
memories, adapt the code accordingly. I'm not interested in doing so.

> tons of systems which, for sane and well-considered reasons, will cap
> allocations far below the size of physical memory or swap.  2*n+1 goes
> pathological pretty fast.  Also, it's... ridiculous, frankly, to start
> at 0.  That guarantees at least one realloc, and probably several for
> even a tiny string.

The total time and space is O(n). Changing the multiplier leaves you with O(n). 
Starting with size>0 reduces the time and space by O(k), however
O(n)-O(k) = O(n). You're worrying about optimising at the wrong level which is a 
waste of time unless you've done measurement showing this is a hot loop. Also 
realloc(0,n) is the same as malloc(n) so one initial malloc is no different than 
a realloc on the first loop; but it requires more code.

> Imagine that you're on a system which, quite reasonably, limits your
> 32-bit code to 3GB of memory.  You can have all 3GB, but you can't have
> more.

I used to write code that had to fit in 65K bytes. I don't any more. You can if 
you want.

> Now imagine that you want to process a string which contains 1GB of
> data.  Your algorithm will do ~30 reallocs, most of which will almost
> certainly be copying all that data, and then fail.  Think it through;
> what happens?  At any given time, your current length is a value of
> the form 2^n - 1.  So what's it going to do for 2^n bytes of data,
> say, 1GB exactly?

What if it's 2^n+1? 2^n+42? 23^n? The nth Fibonacci number? It's still all O(n).

> 1.  It reallocs until it has exactly one byte less than 1GB of data.
> 2.  It then tries to allocate exactly one byte less than 2GB of data,
> to copy that first GB into.
> 
> But that's two bytes less than 3GB of data it needs to be able to handle,
> and I doubt your code and support libraries fit into 2 bytes.

If you really have 1GB lines in text files, or even 1GB text files, adapt the 
code to whatever interface you want. I'm not going to do it unless you pay me.

> Depending, I'd say either start with a "likely" length or start with
> something around a page size, and reallocate up a little more conservatively.

If the resize doesn't increase geometrically, the run time switches to O(n^2). 
Otherwise it is O(n) + whatever code complexity you want to pretend you can beat 
O(n).

-- 
Celle est une langue.     |   With the nutty taste of wild hickory nuts.
C'est francais           /|\              I'm whoever you want me to be.
et tres, tres sexuel.    \|/         Annoying Usenet one post at a time.
- China Azul              |            At least I can stay in character.

[toc] | [prev] | [next] | [standalone]

#11906

In article <chine.bleu-EFDDCF.02510220102011@news.eternal-september.org>,
China Blue Corn Chips  <chine.bleu@yahoo.com> plaintively whind:
...
>I'm not going to do it unless you pay me.

That's what your girlfriends always tell you, isn't it?

-- 
Windows 95 n. (Win-doze): A 32 bit extension to a 16 bit user interface for
an 8 bit operating system based on a 4 bit architecture from a 2 bit company
that can't stand 1 bit of competition.

Modern day upgrade --> Windows XP Professional x64: Windows is now a 64 bit
tweak of a 32 bit extension to a 16 bit user interface for an 8 bit
operating system based on a 4 bit architecture from a 2 bit company that
can't stand 1 bit of competition.

[toc] | [prev] | [next] | [standalone]

#11907

"China Blue Corn Chips" <chine.bleu@yahoo.com> wrote in message 
news:chine.bleu-EFDDCF.02510220102011@news.eternal-september.org...
> In article <slrnj9vjr9.2vs.usenet-nospam@guild.seebs.net>,
> Seebs <usenet-nospam@seebs.net> wrote:
>
>> On 2011-10-20, China Blue Corn Chips <chine.bleu@yahoo.com> wrote:

>> > Buy a bigger swap disk.
>>
>> This is crappy advice, because it's completely unrelated.  There are
>
> The maximum line size is the file size. If the VM that can map in the 
> entire
> file, it can allocate space for one line.

Someone doing 'ReadLine' is probably interested in line-oriented files.

A file containing with a single multi-GB line is likely not line-oriented! 
Or is not using the expected newline sequence.

It would be acceptable then to put a cap on the longest length of one line. 
The user then needs to use a function such as 'Readfile' instead.

-- 
Bartc
 

[toc] | [prev] | [next] | [standalone]

#11911

On Oct 20, 3:31 pm, "BartC" <b...@freeuk.com> wrote:
>
> Someone doing 'ReadLine' is probably interested in line-oriented files.
>
> A file containing with a single multi-GB line is likely not line-oriented!
> Or is not using the expected newline sequence.
>
The problem is that it is impossible to define a reasonable line
length.

For instance a program I've just written generates csv files. Each
line represents a "case', each column a "factor". The factors are DNA
sequences which may or may not be present in the genes under
consideration, the cases are the genes. If I look at 5-letter DNA
sequences, that's 4^5 or 1024 factors. Each one has a real value
associated with it, so about five characters for three decimal places
of precision. That means that lines are weighing in at about 6000
characters.


--
Fuzzy logic trees (build a tree from factors and cases, as described).
http://www.malcolmmclean.site11.com/www

[toc] | [prev] | [next] | [standalone]

#11913

In article <e3302961-696e-4b9f-bc77-9282dc328c93@g7g2000vbv.googlegroups.com>,
Malcolm McLean  <malcolm.mclean5@btinternet.com> wrote:
...
>sequences, that's 4^5 or 1024 factors. Each one has a real value
>associated with it, so about five characters for three decimal places
>of precision. That means that lines are weighing in at about 6000
>characters.

6000 is nothing.  10K is nothing.  Even 1M is really nothing on today's
hardware.

This is one of those issues where in order to make it sound like a serious
problem, people have to drag it out to an extreme (talking about 1G).

-- 

First of all, I do not appreciate your playing stupid here at all.

	- Thomas 'PointedEars' Lahn -

[toc] | [prev] | [next] | [standalone]

#11914

Malcolm McLean <malcolm.mclean5@btinternet.com> writes:

> On Oct 20, 3:31 pm, "BartC" <b...@freeuk.com> wrote:
>>
>> Someone doing 'ReadLine' is probably interested in line-oriented files.
>>
>> A file containing with a single multi-GB line is likely not line-oriented!
>> Or is not using the expected newline sequence.
>>
> The problem is that it is impossible to define a reasonable line
> length.
>
> For instance a program I've just written generates csv files. Each
> line represents a "case', each column a "factor". The factors are DNA
> sequences which may or may not be present in the genes under
> consideration, the cases are the genes. If I look at 5-letter DNA
> sequences, that's 4^5 or 1024 factors. Each one has a real value
> associated with it, so about five characters for three decimal places
> of precision. That means that lines are weighing in at about 6000
> characters.

You say it's impossible, yet you've managed to do it!  Maybe you meant
that there can't be a limit imposed by the function.  I agree that this
is more of a problem, but I don't see why there is any need to do that.
I'd want a general-purpose read_line function to have a maximum size
parameter, and it should probably allow the buffer to be re-used since
that is such a common pattern in line-oriented programs.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#11925

"Malcolm McLean" <malcolm.mclean5@btinternet.com> wrote in message
news:e3302961-696e-4b9f-bc77-9282dc328c93@g7g2000vbv.googlegroups.com...
> On Oct 20, 3:31 pm, "BartC" <b...@freeuk.com> wrote:
>>
>> Someone doing 'ReadLine' is probably interested in line-oriented files.
>>
>> A file containing with a single multi-GB line is likely not
>> line-oriented!
>> Or is not using the expected newline sequence.
>>
> The problem is that it is impossible to define a reasonable line
> length.

A good guide might be how long you are prepared to spend scrolling from one
end to another.

My rough calculations show that a line 30,000 characters long would take
about a minute to scroll from one end to the other (paging a full width,
twice a second). 30,000 characters seems perfectly reasonable (I use 2048
myself and rarely have problems).

(30,000 characters also requires a virtual screen 200 feet wide. A line
buffer 1 billion characters long requires a virtual screen some 1300 miles
across and would take 3 weeks to scroll. No-one can possibly pretend that
that is still one line of text!)

If a limit of a few thousand characters is not enough, then a different
approach is needed; something in between ReadLine and ReadFile; ReadBlock
perhaps, as the file cannot be considered line-oriented, not in the 
conventional, human-readable sense.

-- 
Bartc
 

[toc] | [prev] | [next] | [standalone]

#11926

"BartC" <bc@freeuk.com> writes:
> "Malcolm McLean" <malcolm.mclean5@btinternet.com> wrote in message
> news:e3302961-696e-4b9f-bc77-9282dc328c93@g7g2000vbv.googlegroups.com...
>> On Oct 20, 3:31 pm, "BartC" <b...@freeuk.com> wrote:
>>>
>>> Someone doing 'ReadLine' is probably interested in line-oriented files.
>>>
>>> A file containing with a single multi-GB line is likely not
>>> line-oriented!
>>> Or is not using the expected newline sequence.
>>>
>> The problem is that it is impossible to define a reasonable line
>> length.
>
> A good guide might be how long you are prepared to spend scrolling from one
> end to another.
[snip]
> If a limit of a few thousand characters is not enough, then a different
> approach is needed; something in between ReadLine and ReadFile; ReadBlock
> perhaps, as the file cannot be considered line-oriented, not in the 
> conventional, human-readable sense.

Text isn't necessarily meant to be human-readable.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#11928

On 10/21/11 11:34 AM, Keith Thompson wrote:
> "BartC"<bc@freeuk.com>  writes:
>> "Malcolm McLean"<malcolm.mclean5@btinternet.com>  wrote in message
>> news:e3302961-696e-4b9f-bc77-9282dc328c93@g7g2000vbv.googlegroups.com...
>>> On Oct 20, 3:31 pm, "BartC"<b...@freeuk.com>  wrote:
>>>>
>>>> Someone doing 'ReadLine' is probably interested in line-oriented files.
>>>>
>>>> A file containing with a single multi-GB line is likely not
>>>> line-oriented!
>>>> Or is not using the expected newline sequence.
>>>>
>>> The problem is that it is impossible to define a reasonable line
>>> length.
>>
>> A good guide might be how long you are prepared to spend scrolling from one
>> end to another.
> [snip]
>> If a limit of a few thousand characters is not enough, then a different
>> approach is needed; something in between ReadLine and ReadFile; ReadBlock
>> perhaps, as the file cannot be considered line-oriented, not in the
>> conventional, human-readable sense.
>
> Text isn't necessarily meant to be human-readable.

Indeed.  The longest single "line" I have in a file is 1.3GB (a JSON 
representation of a rather large filesystem).

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#11936

"Keith Thompson" <kst-u@mib.org> wrote in message 
news:lnfwinnvgb.fsf@nuthaus.mib.org...
> "BartC" <bc@freeuk.com> writes:

>> If a limit of a few thousand characters is not enough, then a different
>> approach is needed; something in between ReadLine and ReadFile; ReadBlock
>> perhaps, as the file cannot be considered line-oriented, not in the
>> conventional, human-readable sense.
>
> Text isn't necessarily meant to be human-readable.

In that case there should be no pretence that a ReadLine function actually 
reads a 'line' at all, but any sequence of characters of any length.

That's why I suggested such a function should be called Readblock, with all 
the problems that that would introduce (such as possibly bringing down the 
system in the case of a rogue file with 'lines' too long to fit into either 
physical or virtual memory).

Then 'ReadLine' can be used for the straightforward cases such as reading 
configuration files or command lines. Lines which are too long are reported 
as errors (which in all probability they are).

-- 
Bartc 

[toc] | [prev] | [next] | [standalone]

#11940

On Oct 21, 11:30 am, "BartC" <b...@freeuk.com> wrote:
> "Keith Thompson" <ks...@mib.org> wrote in message
>
> > Text isn't necessarily meant to be human-readable.
>
> In that case there should be no pretence that a ReadLine function actually
> reads a 'line' at all, but any sequence of characters of any length.
>
There's a difference between human-inspectable and human-readable.

In the case of my program, no-one can really make sense of a matrix
with 1000 factors and 6000 cases. However you can certainly look at it
to see that the factors have a reasonable balance of set and missing,
the labels are right, there's the right number of cases (one per
gene), and so on.
--
Visit my website
http://www.malcolmmclean.site11.com/www

[toc] | [prev] | [next] | [standalone]

#11944

"BartC" <bc@freeuk.com> writes:
> "Keith Thompson" <kst-u@mib.org> wrote in message
> news:lnfwinnvgb.fsf@nuthaus.mib.org...
>> "BartC" <bc@freeuk.com> writes:
>
>>> If a limit of a few thousand characters is not enough, then a different
>>> approach is needed; something in between ReadLine and ReadFile; ReadBlock
>>> perhaps, as the file cannot be considered line-oriented, not in the
>>> conventional, human-readable sense.
>>
>> Text isn't necessarily meant to be human-readable.
>
> In that case there should be no pretence that a ReadLine function
> actually reads a 'line' at all, but any sequence of characters of any
> length.
>
> That's why I suggested such a function should be called Readblock,
> with all the problems that that would introduce (such as possibly
> bringing down the system in the case of a rogue file with 'lines' too
> long to fit into either physical or virtual memory).
>
> Then 'ReadLine' can be used for the straightforward cases such as
> reading configuration files or command lines. Lines which are too long
> are reported as errors (which in all probability they are).

A block of characters terminated by a newline character is what we call
a "line".

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#11960

"Keith Thompson" <kst-u@mib.org> wrote in message 
news:ln7h3yns5q.fsf@nuthaus.mib.org...
> "BartC" <bc@freeuk.com> writes:

>> Then 'ReadLine' can be used for the straightforward cases such as
>> reading configuration files or command lines. Lines which are too long
>> are reported as errors (which in all probability they are).
>
> A block of characters terminated by a newline character is what we call
> a "line".

Yes, but is a line a few hundred characters long, as it might have been once 
(eg. 132), or could it conceivably be terrabytes long? I think that in most 
cases where you want to read a line, assuming that the line can be, 
effectively, of infinite length, adds unnecessary complications.

-- 
Bartc
 

[toc] | [prev] | [next] | [standalone]

#11962

On 10/23/11 08:00 AM, BartC wrote:
> "Keith Thompson"<kst-u@mib.org>  wrote in message
> news:ln7h3yns5q.fsf@nuthaus.mib.org...
>> "BartC"<bc@freeuk.com>  writes:
>
>>> Then 'ReadLine' can be used for the straightforward cases such as
>>> reading configuration files or command lines. Lines which are too long
>>> are reported as errors (which in all probability they are).
>>
>> A block of characters terminated by a newline character is what we call
>> a "line".
>
> Yes, but is a line a few hundred characters long, as it might have been once
> (eg. 132), or could it conceivably be terrabytes long? I think that in most
> cases where you want to read a line, assuming that the line can be,
> effectively, of infinite length, adds unnecessary complications.

"where you want to read a line" is the key there, data can be stored in 
an infinitely long "line", but if that data isn't intended to be read as 
one line (my 1.3GB JSON line for example) the argument is moot.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#11974

On Oct 22, 9:00 pm, "BartC" <b...@freeuk.com> wrote:
>
> Yes, but is a line a few hundred characters long, as it might have been once
> (eg. 132), or could it conceivably be terrabytes long? I think that in most
> cases where you want to read a line, assuming that the line can be,
> effectively, of infinite length, adds unnecessary complications.
>
Not really. Library functions make it slightly easier, in C, to read a
line into a fixed buffer. But if you go for an allocated buffer, the
natural thing is to keep expanding until you run out of memory. On
decent systems, behaviour when fed an infinite sequence of characters
is perfectly acceptable.
--
Basic Algorithms: now an Apple ibook
http://www.malcolmmclean.site11.com/www

[toc] | [prev] | [next] | [standalone]

#11921

On 10/21/11 02:31 AM, BartC wrote:
> "China Blue Corn Chips"<chine.bleu@yahoo.com>  wrote in message
> news:chine.bleu-EFDDCF.02510220102011@news.eternal-september.org...
>> In article<slrnj9vjr9.2vs.usenet-nospam@guild.seebs.net>,
>> Seebs<usenet-nospam@seebs.net>  wrote:
>>
>>> On 2011-10-20, China Blue Corn Chips<chine.bleu@yahoo.com>  wrote:
>
>>>> Buy a bigger swap disk.
>>>
>>> This is crappy advice, because it's completely unrelated.  There are
>>
>> The maximum line size is the file size. If the VM that can map in the
>> entire
>> file, it can allocate space for one line.
>
> Someone doing 'ReadLine' is probably interested in line-oriented files.
>
> A file containing with a single multi-GB line is likely not line-oriented!
> Or is not using the expected newline sequence.
>
> It would be acceptable then to put a cap on the longest length of one line.
> The user then needs to use a function such as 'Readfile' instead.

Or consume the "line" bit by bit.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.c

csiph-web