Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.sys.acorn.programmer > #2002 > unrolled thread

Reading text files with cr / lf line terminators

Back to article view | Back to comp.sys.acorn.programmer

Contents

  Reading text files with cr / lf line terminators Dave Stratford <daves@orpheusmail.co.uk> - 2012-08-07 09:05 +0100
    Re: Reading text files with cr / lf line terminators Vince M Hudd <vinceh@softrock.co.uk> - 2012-08-07 12:11 +0100
      Re: Reading text files with cr / lf line terminators Vince M Hudd <vinceh@softrock.co.uk> - 2012-08-07 12:26 +0100
        Re: Reading text files with cr / lf line terminators Dave Stratford <daves@orpheusmail.co.uk> - 2012-08-07 14:52 +0100
    Re: Reading text files with cr / lf line terminators Ron <beeb@woosh.co.nz> - 2012-08-07 23:35 +1200
      Re: Reading text files with cr / lf line terminators Alan Adams <alan@adamshome.org.uk> - 2012-08-07 14:21 +0100
        Re: Reading text files with cr / lf line terminators Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk> - 2012-08-07 15:09 +0100
        Re: Reading text files with cr / lf line terminators Ron <beeb@woosh.co.nz> - 2012-08-08 03:01 +1200
          Re: Reading text files with cr / lf line terminators Ron <beeb@woosh.co.nz> - 2012-08-08 04:13 +1200
    Re: Reading text files with cr / lf line terminators Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk> - 2012-08-07 15:20 +0100
    Re: Reading text files with cr / lf line terminators jgh@arcade.demon.co.uk (Jonathan Graham Harston) - 2012-08-08 22:41 +0100

#2002 — Reading text files with cr / lf line terminators

Hi folks,

have a small basic prog that reads text files. I use line$=GET$#file% to
read each line.

This is okay on RiscOS generated files, but when I read a PC/Dos created
file where each line ends in crlf I end up getting two lines, one with
data, one blank.

A quick and dirty check showed me that GET$# can use both cr and lf as
line terminators, so when they are both there it sees two lines.

What's the best way of trapping this instance, particularly as it's
possible that there will sometimes be genuinely valid blank lines.

I'm guessing that I either read the whole file into a block of memory and
frig it using indirection operators, or I read it byte at a time and try
and determine from that.

Problem with any of these techniques of course is when the line
terminators are not consistent.

Any best practices?

Many thanks,

Dave

-- 
    _                     _________________________________________
   / \._._ |_  _     _  /'       Orpheus Internet Services
   \_/| |_)| |(/_|_|_> /          'Internet for Everyone'
_______ | ___________./       http://www.orpheusinternet.co.uk

[toc] | [next] | [standalone]

#2003

Dave Stratford <daves@orpheusmail.co.uk> wrote:

> have a small basic prog that reads text files. I use line$=GET$#file% to
> read each line.
 
> This is okay on RiscOS generated files, but when I read a PC/Dos created
> file where each line ends in crlf I end up getting two lines, one with
> data, one blank.
 
> A quick and dirty check showed me that GET$# can use both cr and lf as
> line terminators, so when they are both there it sees two lines.

There probably as many ways around this as there are programmers still
actively working on RISC OS applications. :) If it was my code, I'd probably
just hook into the code used in WebChange to change line endings - but it's
not my code, and you're using BASIC.

I'm not at a RISC OS machine to try anything (so the usual usenet tradition
of untested etc applies) but a quick solution that might allow you to
minimise the changes you need to make could be to flag the terminator type
for each line you read. If the line is blank, look at the flag for the
previous line to see what its terminator was. If it's the same, it's a
genuine blank line, if it's different, that implies you have a file with
CR/LF terminators, so this is the second part of the terminator.

I don't think you'll be able to get the terminator from the actual line
returned by GET$, but you should be able to read it from the file, so you
could probably do something like:

----8<----
REM Set the old terminator to that of the last line read
REM Don't forget to initialise these sensibly.
oldterm% = newterm%

REM [re]initialise this variable on each pass
validblank% = TRUE

REM Your line to read a line from the file
line$ = GET$#file%

REM Go back one byte in the file so we're pointing at the terminator
REM that was recognised by GET$
PTR#file% = (PTR#file%)-1

REM Read that terminator
newterm% = BGET#file% 


REM Check if GET$ returned a blank line
IF line$ = "" THEN

  REM and if it did, then whether it's a valid one, or the
  REM result of a mixed terminator in the file

  IF newterm% <> oldterm% THEN validblank% = FALSE

ENDIF
----8<----

Now, along with line$ containing your line, and validblank% contains either
TRUE or FALSE to indicate whether line$ is a valid blank line (if it's a
null string).

So when you're doing whatever you do with the line, when you find it's blank
and need to know if it's a real blank line or the result of a mixed
terminator, just check if validblank contains TRUE or FALSE.

-- 
Soft Rock Software:                   http://www.softrock.co.uk
Vince M Hudd:                         http://misc.vinceh.com/about-vinceh/
RISCOSitory:                          http://www.riscository.com

[toc] | [prev] | [next] | [standalone]

#2004

Vince M Hudd <vinceh@softrock.co.uk> wrote:
> Dave Stratford <daves@orpheusmail.co.uk> wrote:

[...]

> I don't think you'll be able to get the terminator from the actual line
> returned by GET$, but you should be able to read it from the file, so you
> could probably do something like:

Typical, you post, and it isn't until you read your own post when you think
"D'oh!"

Why bother checking the line ending on every line? Why not just check the
line ending (and that of the previous line) when a blank line is
encountered?

----8<----
REM Your line to read a line from the file
line$ = GET$#file%

REM [re]initialise validblank% every time

REM If it's blank, check if it's validly so
IF line$ = "" THEN validblank% = FN_IsItReallyBlank(file%)


REM rest of your code, and somewhere, this function
DEF FN_IsItReallyBlank(handle%)
LOCAL oldterm%, newterm%

REM Go back two bytes in the file
PTR#handle% = (PTR#handle%)-2

REM Read those two bytes as the old terminator and new terminator
oldterm%=BGET#Handle%
newterm%=BGET#Handle%

REM Compare them, return TRUE if they're the same, FALSE if not
IF oldterm%=newterm% THEN =TRUE ELSE =FALSE
----8<----

The outcome is the same - you have a variable, validblank% containing TRUE
if a blank line is a valid one, FALSE if it's the result of a mixed
terminator. The advantage is far fewer additional byte reads from the file
when the file only has single line terminators.

(Again, untested in the usual tradition of usenet, etc)

-- 
Soft Rock Software:                   http://www.softrock.co.uk
Vince M Hudd:                         http://misc.vinceh.com/about-vinceh/
RISCOSitory:                          http://www.riscository.com

[toc] | [prev] | [next] | [standalone]

#2007

In article <mpro.m8dugr003qcq900zk.vinceh@softrock.co.uk>,
   Vince M Hudd <vinceh@softrock.co.uk> wrote:
> Vince M Hudd <vinceh@softrock.co.uk> wrote:
> > Dave Stratford <daves@orpheusmail.co.uk> wrote:

> [...]

> > I don't think you'll be able to get the terminator from the actual line
> > returned by GET$, but you should be able to read it from the file, so you
> > could probably do something like:

> Typical, you post, and it isn't until you read your own post when you think
> "D'oh!"

> Why bother checking the line ending on every line? Why not just check the
> line ending (and that of the previous line) when a blank line is
> encountered?

> ----8<----
> REM Your line to read a line from the file
> line$ = GET$#file%

> REM [re]initialise validblank% every time

> REM If it's blank, check if it's validly so
> IF line$ = "" THEN validblank% = FN_IsItReallyBlank(file%)


> REM rest of your code, and somewhere, this function
> DEF FN_IsItReallyBlank(handle%)
> LOCAL oldterm%, newterm%

> REM Go back two bytes in the file
> PTR#handle% = (PTR#handle%)-2

> REM Read those two bytes as the old terminator and new terminator
> oldterm%=BGET#Handle%
> newterm%=BGET#Handle%

> REM Compare them, return TRUE if they're the same, FALSE if not
> IF oldterm%=newterm% THEN =TRUE ELSE =FALSE
> ----8<----

> The outcome is the same - you have a variable, validblank% containing TRUE
> if a blank line is a valid one, FALSE if it's the result of a mixed
> terminator. The advantage is far fewer additional byte reads from the file
> when the file only has single line terminators.

> (Again, untested in the usual tradition of usenet, etc)

By itself this doesn't work, as it wouldn't catch validly empty blank
lines with a PC/DOS style CRLF line terminator, but with a small tweak it
worked fine:

=== 8< ===

DEF FN_CheckBlank(handle%)
LOCAL oldterm%, newterm%, prev1%, prev2%, curr1%, curr2%

REM Go back two bytes in the file
PTR#handle% = (PTR#handle%)-2

REM Read those two bytes as the old terminator and new terminator
oldterm%=BGET#handle%
newterm%=BGET#handle%

IF (oldterm%=10 AND newterm%=13) OR (oldterm%=13 AND newterm%=10) THEN
REM here we have a cr/lf terminator, so go back another two characters to
see if they too were a cr/lf terminator for a pc/dos style end of line.
PTR#handle% = (PTR#handle%)-4

  prev1%=BGET#handle%
  prev2%=BGET#handle%
  curr1%=BGET#handle%
  curr2%=BGET#handle%

  IF prev1% = curr1% AND prev2% = curr2% THEN =TRUE ELSE =FALSE
ELSE
  IF oldterm%=newterm% THEN =TRUE ELSE =FALSE
ENDIF

=== 8< ===

Many thanks,

Dave


-- 
    _                     _________________________________________
   / \._._ |_  _     _  /'       Orpheus Internet Services
   \_/| |_)| |(/_|_|_> /          'Internet for Everyone'
_______ | ___________./       http://www.orpheusinternet.co.uk

[toc] | [prev] | [next] | [standalone]

#2005

In message <52bb693e1fdaves@orpheusmail.co.uk>
          Dave Stratford <daves@orpheusmail.co.uk> wrote:

> Hi folks,
> 
> have a small basic prog that reads text files. I use line$=GET$#file% to
> read each line.
> 
> This is okay on RiscOS generated files, but when I read a PC/Dos created
> file where each line ends in crlf I end up getting two lines, one with
> data, one blank.
> 
> A quick and dirty check showed me that GET$# can use both cr and lf as
> line terminators, so when they are both there it sees two lines.
> 
> What's the best way of trapping this instance, particularly as it's
> possible that there will sometimes be genuinely valid blank lines.
> 
> I'm guessing that I either read the whole file into a block of memory and
> frig it using indirection operators, or I read it byte at a time and try
> and determine from that.
> 
> Problem with any of these techniques of course is when the line
> terminators are not consistent.
> 

Yes, then what would be a simple job becomes something else.
The indirection operator only recognises CHR$(13) as the terminator
for a string, if you can rely on that.

I've been playing with serial terminals and there is a variation
of line terminations there also.
Reading a character at a time, I guess a flag will need to be set
to check the next character for the existence of a second terminator.

Ron M. 

[toc] | [prev] | [next] | [standalone]

#2006

In message <13697cbb52.beeb@ron1954.woosh.co.nz>
          Ron <beeb@woosh.co.nz> wrote:

> In message <52bb693e1fdaves@orpheusmail.co.uk>
>           Dave Stratford <daves@orpheusmail.co.uk> wrote:

>> Hi folks,
>> 
>> have a small basic prog that reads text files. I use line$=GET$#file% to
>> read each line.
>> 
>> This is okay on RiscOS generated files, but when I read a PC/Dos created
>> file where each line ends in crlf I end up getting two lines, one with
>> data, one blank.
>> 
>> A quick and dirty check showed me that GET$# can use both cr and lf as
>> line terminators, so when they are both there it sees two lines.
>> 
>> What's the best way of trapping this instance, particularly as it's
>> possible that there will sometimes be genuinely valid blank lines.
>> 
>> I'm guessing that I either read the whole file into a block of memory and
>> frig it using indirection operators, or I read it byte at a time and try
>> and determine from that.
>> 
>> Problem with any of these techniques of course is when the line
>> terminators are not consistent.
>> 

> Yes, then what would be a simple job becomes something else.
> The indirection operator only recognises CHR$(13) as the terminator
> for a string, if you can rely on that.

> I've been playing with serial terminals and there is a variation
> of line terminations there also.
> Reading a character at a time, I guess a flag will need to be set
> to check the next character for the existence of a second terminator.

> Ron M.

I'd try a different approach:

Create a large buffer e.g. DIM buffer% 10000

Use OS_GBPB to read a chunk of the file into it

Scan for the first CR or LF, and check whether it is on its own, or a 
pair. You now know what the terminator sequence is, i.e. LF, or CR or 
CRLF or LFCR

Now you can scan the buffer for this combination, and extract 
everything between as lines. When you get to the end of the buffer 
it's unlikely you will also have a terminator, but you might.

Read the remaining bytes somewhere (e.g. into a string), then refill 
the buffer.
Find the first terminator sequence, and append what precedes it to 
your termorary area. Then carry on as before until EOF.

(Check the returned bytes from OS_GBPB each time - if it is less than 
the buffer size, you have reached EOF. You can also check EOF#handle)

You might need to deal with a CRLF split between buffers - check if 
the last character in your buffer is the beginning of a terminator 
sequence.

You could make it a bit slicker by using a ring buffer - you keep 
start and end of data pointers. When you have a bit left in the 
buffer, you refill it up to just before the last valid data, then read 
round the end to the start. It's a bit more tricky to code, but partly 
gets rid of the special cases at the end of the buffer.

Alan


-- 
Alan Adams, from Northamptonshire
alan@adamshome.org.uk
http://www.nckc.org.uk/

[toc] | [prev] | [next] | [standalone]

#2008

Alan Adams <alan@adamshome.org.uk> wrote:

> Scan for the first CR or LF, and check whether it is on its own, or a
> pair. You now know what the terminator sequence is, i.e. LF, or CR or CRLF
> or LFCR

I'm not sure that I'd stop having found the first CR or LF, because that
might exist on its own in the midst of the first 'line' of the file, with
the true line delimiter first being seen later on.

I'd be more inclined to count the number of occurrences of just LF, just CR,
LF CR, and CR LF in the test chunk, and then try to decide if any one set
was dominant. 



-- 
Jeremy C B Nicoll - my opinions are my own.

Email sent to my from-address will be deleted. Instead, please reply
to newsreplyaaa@wingsandbeaks.org.uk replacing "aaa" by "284".  

[toc] | [prev] | [next] | [standalone]

#2010

cn message <012886bb52.Alan.Adams@laptop.adamshome.org.uk>
          Alan Adams <alan@adamshome.org.uk> wrote:

> In message <13697cbb52.beeb@ron1954.woosh.co.nz>
>           Ron <beeb@woosh.co.nz> wrote:
> 

> 
> > Yes, then what would be a simple job becomes something else.
> > The indirection operator only recognises CHR$(13) as the terminator
> > for a string, if you can rely on that.
> 
> > I've been playing with serial terminals and there is a variation
> > of line terminations there also.
> > Reading a character at a time, I guess a flag will need to be set
> > to check the next character for the existence of a second terminator.
> 
> > Ron M.
> 
> I'd try a different approach:
> 
> Create a large buffer e.g. DIM buffer% 10000
> 
> Use OS_GBPB to read a chunk of the file into it
> 
<snip>

Yes but for character editing, I am doing it character at a time,
For example, typing a command for the CLI from a remote computer
then allows backspace corrections before hitting Enter.

To print a single line feed for all CR/LF combinations
I need something functionally like:

PROCprintchar
IF char$<>CHR$(10) THEN IF char$<>CHR$(13) term%=0
IF term%=0 PRINT char$;
IF char$=CHR$(13) THEN IF term%=0 PRINT 
IF char$=CHR$(13) term%=1
IF char$=CHR$(10) term%=1
ENDPROC

The last two lines ensure that only one PRINT(or LF) happens,
and the first line resets the flag if back to non termination
characters.

[toc] | [prev] | [next] | [standalone]

#2012

In message <824f8fbb52.beeb@ron1954.woosh.co.nz>
          Ron <beeb@woosh.co.nz> wrote:
<snip>
> To print a single line feed for all CR/LF combinations
> I need something functionally like:
> 
> PROCprintchar
> IF char$<>CHR$(10) THEN IF char$<>CHR$(13) term%=0
> IF term%=0 PRINT char$;
> IF char$=CHR$(13) THEN IF term%=0 PRINT 
> IF char$=CHR$(13) term%=1
> IF char$=CHR$(10) term%=1
> ENDPROC
> 
> The last two lines ensure that only one PRINT(or LF) happens,
> and the first line resets the flag if back to non termination
> characters.

To allow two or more (intentional)lf/cr's through it might be

PROCprintchar
IF char$<>CHR$(10) THEN IF char$<>CHR$(13) term%=0
IF term%=0 PRINT char$;
IF char$=CHR$(13) THEN IF term%<>2 PRINT
IF char$=CHR$(10) THEN IF term%<>1 PRINT 
IF char$=CHR$(13) THEN IF term%<>2 term%=1
IF char$=CHR$(10) THEN IF term%<>1 term%=2
ENDPROC

I think that will print CRCRCR or LFLFLF and only use the first
char of CRLFCRLFCRLF  or LFCRLFCRLFCR  type pairs

Ron M

[toc] | [prev] | [next] | [standalone]

#2009

Dave Stratford <daves@orpheusmail.co.uk> wrote:

> This is okay on RiscOS generated files...

It might be ok on test files generated by, say !Edit, but RO also generates
text files with other line-end delimiters in some situation that I can't
quite remember... 

It might be files put together with  *echo adfjkdfjhd { > somefile }

or files concatenated together using *print and redirection.  I'm sorry, I
can't remember... 

-- 
Jeremy C B Nicoll - my opinions are my own.

Email sent to my from-address will be deleted. Instead, please reply
to newsreplyaaa@wingsandbeaks.org.uk replacing "aaa" by "284".  

[toc] | [prev] | [next] | [standalone]

#2036

daves wrote:
> This is okay on RiscOS generated files, but when I read a PC/Dos created
> file where each line ends in crlf I end up getting two lines, one with
> data, one blank.
...
> What's the best way of trapping this instance, particularly as it's
> possible that there will sometimes be genuinely valid blank lines.
 
http://mdfs.net/blib/StringIO is the cannonical method.
 
Credit to Richard Russell for the original version.
 
-- 
J.G.Harston - jgh@mdfs.net - mdfs.net/jgh
Whitby Yards Gazetteer - http://mdfs.net/Docs/Books/YofWhitby/Gazetteer

[toc] | [prev] | [standalone]

Back to top | Article view | comp.sys.acorn.programmer

csiph-web