Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #72234 > unrolled thread

Working with files on different operating systems

Back to article view | Back to comp.lang.c

Contents

  Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 05:11 +0200
    Re: Working with files on different operating systems Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2015-10-08 22:12 -0600
    Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 00:18 -0500
      Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-09 11:13 +0100
        Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 18:05 +0200
          Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 12:49 -0500
          Re: Working with files on different operating systems Eric Sosman <esosman@comcast-dot-net.invalid> - 2015-10-09 17:32 -0400
          Re: Working with files on different operating systems Ian Collins <ian-news@hotmail.com> - 2015-10-10 12:37 +1300
        Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 12:48 -0500
          Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-09 19:53 +0100
            Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 21:09 +0200
              Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-09 20:28 +0100
                Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 21:40 +0200
                  Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-09 15:59 -0400
                    Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 22:07 +0200
                      Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-09 16:24 -0400
                        Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-10 03:08 +0200
                  Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-09 21:50 +0100
          Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-09 14:54 -0400
            Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 21:05 +0200
              Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 21:47 +0200
              Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-09 15:53 -0400
                Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 22:01 +0200
                  Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-09 16:05 -0400
            Re: Working with files on different operating systems glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-10-10 01:16 +0000
              Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-10 03:46 +0200
              Re: Working with files on different operating systems Eric Sosman <esosman@comcast-dot-net.invalid> - 2015-10-10 08:08 -0400
            Re: Working with files on different operating systems raltbos@xs4all.nl (Richard Bos) - 2015-10-10 17:02 +0000
          Re: Working with files on different operating systems Eric Sosman <esosman@comcast-dot-net.invalid> - 2015-10-09 15:53 -0400
      Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 18:00 +0200
        Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 12:59 -0500
          Re: Working with files on different operating systems Keith Thompson <kst-u@mib.org> - 2015-10-09 12:37 -0700
            Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 17:36 -0500
              Re: Working with files on different operating systems Keith Thompson <kst-u@mib.org> - 2015-10-09 16:52 -0700
                Re: Working with files on different operating systems raltbos@xs4all.nl (Richard Bos) - 2015-10-14 11:27 +0000
                  Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-14 11:27 -0400
                  Re: Working with files on different operating systems Jorgen Grahn <grahn+nntp@snipabacken.se> - 2015-10-14 16:46 +0000
                  Re: Working with files on different operating systems Keith Thompson <kst-u@mib.org> - 2015-10-14 10:18 -0700
                Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-14 23:06 -0500
      Re: Working with files on different operating systems Keith Thompson <kst-u@mib.org> - 2015-10-09 09:16 -0700
        Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 17:43 -0500
          Re: Working with files on different operating systems Keith Thompson <kst-u@mib.org> - 2015-10-09 16:57 -0700
            Re: Working with files on different operating systems Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-10-09 22:41 -0700
      Re: Working with files on different operating systems David Thompson <dave.thompson2@verizon.net> - 2015-10-31 06:50 -0400
    Re: Working with files on different operating systems Siri Cruz <chine.bleu@yahoo.com> - 2015-10-08 23:25 -0700
      Re: Working with files on different operating systems Geoff <geoff@invalid.invalid> - 2015-10-10 10:35 -0700
    Re: Working with files on different operating systems Eric Sosman <esosman@comcast-dot-net.invalid> - 2015-10-09 08:42 -0400
      Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 18:51 +0200
        Re: Working with files on different operating systems Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-10-15 18:09 -0700
          Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-17 04:01 +0200
            Re: Working with files on different operating systems Ian Collins <ian-news@hotmail.com> - 2015-10-17 15:55 +1300
              Re: Working with files on different operating systems Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-10-16 20:09 -0700
                Re: Working with files on different operating systems Ian Collins <ian-news@hotmail.com> - 2015-10-17 16:36 +1300
              Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-17 05:10 +0200
          Re: Working with files on different operating systems supercat@casperkitty.com - 2015-10-31 12:05 -0700
            Re: Working with files on different operating systems Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-10-31 12:42 -0700
              Re: Working with files on different operating systems supercat@casperkitty.com - 2015-10-31 13:40 -0700
            Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-31 19:55 +0000
              Re: Working with files on different operating systems supercat@casperkitty.com - 2015-10-31 13:43 -0700
                Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-31 21:38 +0000

Page 3 of 3 — ← Prev page 1 2 [3]

#72342

Keith Thompson wrote:
> Les Cargill <lcargill99@comcast.com> writes:
>> Morten W. Petersen wrote:
>>> I was working on removing warnings from 'gcc -Wall....' and found that
>>> fileno was a POSIX-specific function call,
>>
>> ???
>>
>> For Mingw:
>>
>> C:\windows\system32>grep fileno \MinGW\include\*.h
>> \MinGW\include\stdio.h:_CRTIMP int __cdecl __MINGW_NOTHROW      _fileno
>> (FILE*);
>> \MinGW\include\stdio.h:_CRTIMP int __cdecl __MINGW_NOTHROW      fileno
>> (FILE*);
>> \MinGW\include\stdio.h:#define _fileno(__F) ((__F)->_file)
>> \MinGW\include\stdio.h:#define fileno(__F) ((__F)->_file)
>>
>> C:\windows\system32>
>>
>>
>> It's part of the stdio.h ... canon, which, SFAIK well
>> antedates and has no dependencies on posix.
>
> The fileno() function is not defined by the ISO C standard.  It is
> defined by POSIX.  It's declared in <stdio.h> for historical reasons
> that probably predate the C and POSIX standards, but the C standard
> committee chose not to standardize it (or any functions that work with
> POSIX-style file descriptors).
>
> (IMHO it would have been better if all POSIX-specific declarations
> were separate headers not defined by ISO C.)
>

Interesting - I stand corrected. I had rounded "it precedes POSIX"
to "it's not POSIX."

> A conforming ISO C implementation may not unconditionally define the
> fileno() function in <stdio.h>, since a strictly conforming program is
> free to use that identifier for its own purposes.  A conforming POSIX
> implementation must do so.  Which means that, in order to use
> POSIX-specific functions, you have to invoke the compiler in a
> non-ISO-C-conforming mode (perhaps by defining a macro and/or using a
> particular command-line option).
>
>>> and that finding the file
>>> size on Linux and Windows were two different procedures.
>>
>> This should also not be.
>
> And yet it is.
>

Not if you use certain disciplines...

> [...]
>
>> Your original plan should be just fine. This is the way I have
>> determined file sizes since the 20th Century, modulo declariong
>> and initializing things at the same time for some cruder toolchains:
>>
>> FILE *f = fopen(....);
>>
>> int rc = fseek(f,SEEK_END,0L);
>> int len = ftell(f);
>> rc = fseek(f,SEEK_SET,0L);
>>
>> after which, the file is nicely rewound and len contains the total
>> number of octets in the file.
>>
>> Obviously  for some file systems this may not work, but those are
>> hard to find now.
>
> As we've discussed here recently, the C standard does not guarantee
> that that will work.

Oh, absolutely - I was speaking in very broad terms and not strictly
in terms of the standard.

> The value returned by ftell() is not guaranteed
> to be meaningful, and neither fseek() nor ftell() is guaranteed to
> work at all on all files.  On Windows, the size of a file generally
> does not match the number of characters that can be read from it
> in text mode.
>

This is an excellent reason not to use text mode. Text mode also
can be related to, or ( on the right platforms ) may even use directly 
"cooked mode" from the old BIOS calls, and that's just wrong :)

> POSIX systems have the stat() and fstat() functions, which will give
> you the size of a file, among other information (I think they'll
> typically report 0 for special files that don't have a size).
> Windows has its own function(s) for doing the same kind of thing.
> (And yes, those functions will usually give you the same result as
> the fseek/ftell method.)
>
> Any method for determining the size of a file can only tell you what
> the size was when you checked it.

Yep.

>  A file can grow, shrink, become
> inaccessible, or cease to exist between the size determination and
> a subsequent attempt to read it.  Which doesn't mean that trying
> determine its size is useless, but code to access the file should
> be robust in the presence of possible changes.

Yes sir. Check the return codes.

>  It's sensible to
> use the apparent size as an *estimate* of how much memory you'll
> need to store its contents.
>

-- 
Les Cargill

[toc] | [prev] | [next] | [standalone]

#72355

Les Cargill <lcargill99@comcast.com> writes:
> Keith Thompson wrote:
[Snipping stuff where we pretty much agree]
>> The value returned by ftell() is not guaranteed
>> to be meaningful, and neither fseek() nor ftell() is guaranteed to
>> work at all on all files.  On Windows, the size of a file generally
>> does not match the number of characters that can be read from it
>> in text mode.
>
> This is an excellent reason not to use text mode. Text mode also
> can be related to, or ( on the right platforms ) may even use directly 
> "cooked mode" from the old BIOS calls, and that's just wrong :)

Rather, I'd say it's an excellent reason not to assume that I can
predict exactly how many bytes I'll be able to read from a file in
text mode.

I'd rather use text mode than deal with the vagaries of "\r\n"
line endings and Ctrl-Z end-of-file markers myself.

If I need to slurp an entire text file into memory, and my code
only needs to be portable to POSIX and Windows systems, I think I
can assume that the size of the file (determined by whatever means
seem appropriate) is an upper bound on the number of bytes I'll
read (if I can also assume the file hasn't changed).  The file size
should at least be a good first approximation of how much data I
can read from it.  Of course, *always* check results and be prepared
for surprises.

And I'll acknowledge that there are times when slurping an entire
file into memory is a sensible thing to do (and times when it
definitely isn't).

[...]

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#72371

On Saturday, October 10, 2015 at 12:58:02 AM UTC+1, Keith Thompson wrote:
> Les Cargill <lcargill99@comcast.com> writes:
> 
> And I'll acknowledge that there are times when slurping an entire
> file into memory is a sensible thing to do (and times when it
> definitely isn't).
> 
When it is, normally the file is small. Minibasic reads a basic
source file into memory, then interprets it. If the basic script is
human-generated, it's unlikely to go above a few kilobytes.
So whilst you could get a bit of a speed up from knowing the file
size before allocating the buffer, it's unlikely to be significant.
 

[toc] | [prev] | [next] | [standalone]

#74420

On Fri, 9 Oct 2015 00:18:32 -0500, Les Cargill
<lcargill99@comcast.com> wrote:

> Morten W. Petersen wrote:

> > I was working on removing warnings from 'gcc -Wall....' and found that
> > fileno was a POSIX-specific function call,
> 
> ???
> 
> For Mingw:
> 
> C:\windows\system32>grep fileno \MinGW\include\*.h
> \MinGW\include\stdio.h:_CRTIMP int __cdecl __MINGW_NOTHROW      _fileno 
> (FILE*);
> \MinGW\include\stdio.h:_CRTIMP int __cdecl __MINGW_NOTHROW      fileno 
> (FILE*);
> \MinGW\include\stdio.h:#define _fileno(__F) ((__F)->_file)
> \MinGW\include\stdio.h:#define fileno(__F) ((__F)->_file)
> 
Look a few lines before and after and you'll see those lines are under
a comment "Other Non ANSI Functions" and are only compiled 
#ifndef __STRICT_ANSI__ 

> It's part of the stdio.h ... canon, which, SFAIK well
> antedates and has no dependencies on posix.
> 
It was in stdio on Unix, and POSIX is the standardized form of Unix.
Along with the rest of Unix's "low-level I/O" (open, read, write,
seek, fcntl, ioctl, close) it went into POSIX but NOT Standard C, 
because a major point of Standard C is to work on NON-Unix.

> > and that finding the file
> > size on Linux and Windows were two different procedures.
> >
> 
> This should also not be.
> 
Windows I/O for "regular" files is almost the same as Unix or POSIX. 
For files that happen to contain text, Windows normally has CRLF 
and Unix/POSIX normally don't, but at "low" level that's ignored. 
OTOH directories, pipes, sockets, and devices are different. So are
processes and threads, but those mostly don't implicate I/O.

> > The file size is used to allocate a buffer that will hold the file
> > contents, but after some thinking, it might as well be a good idea
> > to work on something that accepts a FILE argument and reads in
> > increments on that.
> >
> > That way, the code for reading a file and reading a stream for example
> > from the network is the same.
> >
> 
> Are you familiar with select/poll/epoll? Might be time to
> learn about them; this is a good ( albeit tedious ) way to
> unify these things.
> 
And here's a difference: select etc on Unix covers all fds;
[wsa]select on Windows only handles sockets not other files.
And Unix read and write (and close) handle both sockets and disk files
(and more), but Windows doesn't.

> > It might be enough to realloc, but maybe a bit more robust solution
> > for adding onto a buffer in increments is to have a buffer segment
> > that is linked to the next buffer segment and so on.
> >
> > Any thoughts on this?
> >
> 
> Your original plan should be just fine. This is the way I have 
> determined file sizes since the 20th Century, modulo declariong
> and initializing things at the same time for some cruder toolchains:
> 
> FILE *f = fopen(....);
> 
> int rc = fseek(f,SEEK_END,0L);
> int len = ftell(f);
> rc = fseek(f,SEEK_SET,0L);
> 
Other way: fseek (f, 0L, SEEK_END) etc.

And ftell()'s return is long. Some implementations have long wider
than int -- and support files whose size doesn't fit in 32-bit int.

Of course, that's no guarantee that malloc can allocate that much, or
even that size_t passed to malloc can represent it correctly.

> after which, the file is nicely rewound and len contains the total 
> number of octets in the file.
> 
> Obviously  for some file systems this may not work, but those are
> hard to find now.
> 

[toc] | [prev] | [next] | [standalone]

#72238

In article <mv7b7l$gru$1@speranza.aioe.org>,
 "Morten W. Petersen" <morphex@gmail.com> wrote:

> It might be enough to realloc, but maybe a bit more robust solution
> for adding onto a buffer in increments is to have a buffer segment
> that is linked to the next buffer segment and so on.

char *image = 0; {
    size_t m = 0, n = 0; int c; char *image = 0;
    while ((c=fgetc(file))!=EOF) {
        if (n>=m) {m = 2*(n+1); image = realloc(image, m);}
        image[n++] = c;
    }
    if (n>0) {image = realloc(image, n+1); image[n] = 0;}
}

-- 
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted.
'I desire mercy, not sacrifice.'
When is a Kenyan not a Kenyan? When he's a Canadian.
That's People's Commissioner Siri Cruz now. Punch!

[toc] | [prev] | [next] | [standalone]

#72401

On Thu, 08 Oct 2015 23:25:54 -0700, Siri Cruz <chine.bleu@yahoo.com>
wrote:

>char *image = 0; {
>    size_t m = 0, n = 0; int c; char *image = 0;
>    while ((c=fgetc(file))!=EOF) {
>        if (n>=m) {m = 2*(n+1); image = realloc(image, m);}
>        image[n++] = c;
>    }
>    if (n>0) {image = realloc(image, n+1); image[n] = 0;}
>}

So much badness on so many levels.

[toc] | [prev] | [next] | [standalone]

#72261

On 10/8/2015 11:11 PM, Morten W. Petersen wrote:
> Hoi.
>
> I was working on removing warnings from 'gcc -Wall....' and found that
> fileno was a POSIX-specific function call, and that finding the file
> size on Linux and Windows were two different procedures.
>
> The file size is used to allocate a buffer that will hold the file
> contents, but after some thinking, it might as well be a good idea
> to work on something that accepts a FILE argument and reads in
> increments on that.
>
> That way, the code for reading a file and reading a stream for example
> from the network is the same.
>
> It might be enough to realloc, but maybe a bit more robust solution
> for adding onto a buffer in increments is to have a buffer segment
> that is linked to the next buffer segment and so on.
>
> Any thoughts on this?

     malloc-read-realloc-read is easy, and traditional. Received wisdom
suggests growing the buffer by a ratio rather than by adding a fixed
amount, but I'm not aware of any Really Solid Grounds for guessing the
size of a file of unknown size.

     A linked list of buffers seems attractive for "large" files, though
it makes some operations more complicated: For example, you can't use
strstr() to search for a pattern that's split across two buffers.  As
before, received wisdom suggests making later buffers bigger than the
earlier ones, and as before I'm unaware of any RSG.

     Finally, ask whether you really need the entire file in memory at
one time.  Sometimes you do, but sometimes you can arrange to process
a chunk, discard it, process the next chunk, discard it, and so on.
Lexers, for example, often need only a limited "window" to recognize
a token -- sometimes the window need be only a character or two wide.

-- 
esosman@comcast-dot-net.invalid
"Don't be afraid of work. Make work afraid of you." -- TLM

[toc] | [prev] | [next] | [standalone]

#72279

On 09.10.2015 14:42, Eric Sosman wrote:
> On 10/8/2015 11:11 PM, Morten W. Petersen wrote:
[...]
>> That way, the code for reading a file and reading a stream for example
>> from the network is the same.
>>
>> It might be enough to realloc, but maybe a bit more robust solution
>> for adding onto a buffer in increments is to have a buffer segment
>> that is linked to the next buffer segment and so on.
>>
>> Any thoughts on this?
>
>      malloc-read-realloc-read is easy, and traditional. Received wisdom
> suggests growing the buffer by a ratio rather than by adding a fixed
> amount, but I'm not aware of any Really Solid Grounds for guessing the
> size of a file of unknown size.
>
>      A linked list of buffers seems attractive for "large" files, though
> it makes some operations more complicated: For example, you can't use
> strstr() to search for a pattern that's split across two buffers.  As
> before, received wisdom suggests making later buffers bigger than the
> earlier ones, and as before I'm unaware of any RSG.
>
>      Finally, ask whether you really need the entire file in memory at
> one time.  Sometimes you do, but sometimes you can arrange to process
> a chunk, discard it, process the next chunk, discard it, and so on.
> Lexers, for example, often need only a limited "window" to recognize
> a token -- sometimes the window need be only a character or two wide.

I like the idea of searching for a string using for example strstr as
I assume that is about as fast as a search can get.

However, that the program can read from any stream and has as little
platform-specific code, are two significant good things..  I think
it is better to go back later and add optimizations.

Same goes for the storage type; incremental reads will make the program
use less memory at peaks which counts towards stability and the ability
to deal with large files.

The "window" will most of the time not be that large, but that doesn't
help if there are huge windows in a given XML stream.  And that's
something that has to be taken into account.

-Morten

[toc] | [prev] | [next] | [standalone]

#73004

On Friday, October 9, 2015 at 5:50:47 PM UTC+1, Morten W. Petersen wrote:
>  
> I like the idea of searching for a string using for example strstr as
> I assume that is about as fast as a search can get.
>
Yes and no.
If you're doing lots of searches over a stable string (e.g. making
an encyclopedia searchable), the fastest option is a suffix tree.
But you've got to build the suffix tree, so it's unlikely that
you can beat strstr() for temporary strings representing XML
as it is loaded in. 
>
> However, that the program can read from any stream and has as little
> platform-specific code, are two significant good things..  I think
> it is better to go back later and add optimizations.
>
Generally it's more important to to portable and flexible than to
be as fast as possible. Sadly, C's FILE *s can't be used in all
situations. Really streams are one of those things which need 
object-based design, a stream that is attached to an unreliable 
connection needs to be able to query the user for timeout,
for example, sometimes the user will quit, other times he'll 
tell it to keep trying and go for a cup of coffee. An XML parser 
shouldn't be concerned with that.
>
> Same goes for the storage type; incremental reads will make the program
> use less memory at peaks which counts towards stability and the ability
> to deal with large files.
> 
> The "window" will most of the time not be that large, but that doesn't
> help if there are huge windows in a given XML stream.  And that's
> something that has to be taken into account.
> 
You can't design for everything.
Is the goal to have really good performance on normal XML, even at
the cost of poor performance or even failure on very abnormal
documents? Or do you want to be able to handle absolutely anything
that is within the capacity of the machine, reasonably quickly,
even if that means the parser is quite a bit slower than competitors
on normal XML?  

[toc] | [prev] | [next] | [standalone]

#73124

On 16.10.2015 03:09, Malcolm McLean wrote:
> On Friday, October 9, 2015 at 5:50:47 PM UTC+1, Morten W. Petersen wrote:
>>
>> I like the idea of searching for a string using for example strstr as
>> I assume that is about as fast as a search can get.
>>
> Yes and no.
> If you're doing lots of searches over a stable string (e.g. making
> an encyclopedia searchable), the fastest option is a suffix tree.
> But you've got to build the suffix tree, so it's unlikely that
> you can beat strstr() for temporary strings representing XML
> as it is loaded in.

OK, about as fast it can get without using special structures and
algorithms. :)

>> However, that the program can read from any stream and has as little
>> platform-specific code, are two significant good things..  I think
>> it is better to go back later and add optimizations.
>>
> Generally it's more important to to portable and flexible than to
> be as fast as possible. Sadly, C's FILE *s can't be used in all
> situations. Really streams are one of those things which need
> object-based design, a stream that is attached to an unreliable
> connection needs to be able to query the user for timeout,
> for example, sometimes the user will quit, other times he'll
> tell it to keep trying and go for a cup of coffee. An XML parser
> shouldn't be concerned with that.

Well, the parser can't do any parsing if it runs out of characters
to work on.  I think threading here can be one of the things that
makes working with streams more practical.

As for design without threads, maybe it is a good idea to allocate
some xml_item structs, compress data and so on while idling for input.

But you raise an interesting point, that the higher-level code or a
user might want to abort the parsing, and that's something that has
to be taken into account as well.

>> Same goes for the storage type; incremental reads will make the program
>> use less memory at peaks which counts towards stability and the ability
>> to deal with large files.
>>
>> The "window" will most of the time not be that large, but that doesn't
>> help if there are huge windows in a given XML stream.  And that's
>> something that has to be taken into account.
>>
> You can't design for everything.
> Is the goal to have really good performance on normal XML, even at
> the cost of poor performance or even failure on very abnormal
> documents? Or do you want to be able to handle absolutely anything
> that is within the capacity of the machine, reasonably quickly,
> even if that means the parser is quite a bit slower than competitors
> on normal XML?

Take for example an XML element attribute name.  It will probably not
be that long in most cases.  But the attribute value could be long,
very long or extremely long.

This content has to be allocated space for, parsed and so on, and if
one starts with a default of maybe 32 Unicode characters and reallocs
that down or takes the size * 1.5 for every increase in storage space,
isn't that an OK way to do it?

I'm prototyping now the print_tree function so that it uses recursion,
but everything that can recurse many times will be made into loops
instead, to avoid stack smashing, as well as giving useable feedback to
the higher level code as to what's wrong.

-Morten

[toc] | [prev] | [next] | [standalone]

#73127

Morten W. Petersen wrote:
> On 16.10.2015 03:09, Malcolm McLean wrote:
>> On Friday, October 9, 2015 at 5:50:47 PM UTC+1, Morten W. Petersen wrote:
>
>>> However, that the program can read from any stream and has as little
>>> platform-specific code, are two significant good things..  I think
>>> it is better to go back later and add optimizations.
>>>
>> Generally it's more important to to portable and flexible than to
>> be as fast as possible. Sadly, C's FILE *s can't be used in all
>> situations. Really streams are one of those things which need
>> object-based design, a stream that is attached to an unreliable
>> connection needs to be able to query the user for timeout,
>> for example, sometimes the user will quit, other times he'll
>> tell it to keep trying and go for a cup of coffee. An XML parser
>> shouldn't be concerned with that.
>
> Well, the parser can't do any parsing if it runs out of characters
> to work on.  I think threading here can be one of the things that
> makes working with streams more practical.

Not really, you can't finish parsing the document until all of the 
document has been received.  Assuming you can process the file faster 
than you can read it, any threads would be idle.

> As for design without threads, maybe it is a good idea to allocate
> some xml_item structs, compress data and so on while idling for input.

Where threads can be of use is for data processing unrelated to the 
parsing activity.  One example is an application I had a while back 
where fields in incoming SOAP messages triggered database queries.

>>> Same goes for the storage type; incremental reads will make the program
>>> use less memory at peaks which counts towards stability and the ability
>>> to deal with large files.
>>>
>>> The "window" will most of the time not be that large, but that doesn't
>>> help if there are huge windows in a given XML stream.  And that's
>>> something that has to be taken into account.
>>>
>> You can't design for everything.
>> Is the goal to have really good performance on normal XML, even at
>> the cost of poor performance or even failure on very abnormal
>> documents? Or do you want to be able to handle absolutely anything
>> that is within the capacity of the machine, reasonably quickly,
>> even if that means the parser is quite a bit slower than competitors
>> on normal XML?
>
> Take for example an XML element attribute name.  It will probably not
> be that long in most cases.  But the attribute value could be long,
> very long or extremely long.

You can't prejudge anything when it comes to XML document structure 
unless you are targeting a specific document type.  For example an Open 
Document XML with usually have attribute names way longer then the 
value!  Attributes such as 'text:display-outline-level="0"' abound.

I work with may different types of XML and I haven't found any 
micro-optimisations worth adding to my parser.  Relative to providing a 
comprehensive DOM model, parsing is a doddle in both code complexity and 
size!
-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#73130

On Saturday, October 17, 2015 at 3:55:39 AM UTC+1, Ian Collins wrote:
>
> Relative to providing a comprehensive DOM model, parsing is a doddle 
> in both code complexity and size!
> 
Can you expand that point a bit?

My vanilla xml parser just uses a simple lisp-style linked list
of nodes. It's fine for the intended use, which is reading short to
medium sized documents where O(N^2) searches through lists of children 
aren't likely to be a serious problem. But it wouldn't stand up to
tens of thousands of elements.

[toc] | [prev] | [next] | [standalone]

#73135

Malcolm McLean wrote:
> On Saturday, October 17, 2015 at 3:55:39 AM UTC+1, Ian Collins wrote:
>>
>> Relative to providing a comprehensive DOM model, parsing is a doddle
>> in both code complexity and size!
>>
> Can you expand that point a bit?
>
> My vanilla xml parser just uses a simple lisp-style linked list
> of nodes. It's fine for the intended use, which is reading short to
> medium sized documents where O(N^2) searches through lists of children
> aren't likely to be a serious problem. But it wouldn't stand up to
> tens of thousands of elements.

I chose to implement nearly all of the DOM Level 3 Core specification :)

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#73132

On 17.10.2015 04:55, Ian Collins wrote:
> Morten W. Petersen wrote:
[...]
>> Well, the parser can't do any parsing if it runs out of characters
>> to work on.  I think threading here can be one of the things that
>> makes working with streams more practical.
>
> Not really, you can't finish parsing the document until all of the
> document has been received.  Assuming you can process the file faster
> than you can read it, any threads would be idle.
>
>> As for design without threads, maybe it is a good idea to allocate
>> some xml_item structs, compress data and so on while idling for input.
>
> Where threads can be of use is for data processing unrelated to the
> parsing activity.  One example is an application I had a while back
> where fields in incoming SOAP messages triggered database queries.

Yep, that's another interesting point, that the calling applications
could be able to process the XML document in parts.

>> Take for example an XML element attribute name.  It will probably not
>> be that long in most cases.  But the attribute value could be long,
>> very long or extremely long.
>
> You can't prejudge anything when it comes to XML document structure
> unless you are targeting a specific document type.  For example an Open
> Document XML with usually have attribute names way longer then the
> value!  Attributes such as 'text:display-outline-level="0"' abound.
>
> I work with may different types of XML and I haven't found any
> micro-optimisations worth adding to my parser.  Relative to providing a
> comprehensive DOM model, parsing is a doddle in both code complexity and
> size!

Some assumptions can and have to be made, but anything that passes
through a given XML validator should be able to pass through the parser.

I was wondering about limits for example for entity names; they have
a restricted set of characters that are allowed, and it is unlikely
that someone in their right mind will choose to encode characters that
are not allowed into an element name to be able to include a large
string, when they can put them in attribute values, element content or
CDATA sections.

-Morten

[toc] | [prev] | [next] | [standalone]

#74473

On Thursday, October 15, 2015 at 8:10:23 PM UTC-5, Malcolm McLean wrote:
> Generally it's more important to to portable and flexible than to
> be as fast as possible. Sadly, C's FILE *s can't be used in all
> situations.

I wonder what the performance implications would have been of making the
first member of "file" be a pointer to a structure containing pointers to
all the various functions a file could support.  That would have made it
possible to pass many kinds of things to code which expects to read from
a FILE*.  A variety of means could then have been used to allow files to
support various "optional" features, e.g. a general purpose method:

    int get_ext_info(FILE *f, void *dest, int dest_size, char *name);

which would allow code to do something like:

    TIMEOUT_CONTRL tc;
    if (get_ext_info(myFile, &tc, sizeof tc, "TIMEOUT_CONTROL") > 0)
      tc->setTimeout(&tc, 100);

String-based name lookups wouldn't be wonderful, but in most sorts of file
classes wouldn't support many names, and in many cases the lookup would
only be needed once per file and code could use cached results after that.

[toc] | [prev] | [next] | [standalone]

#74479

On Saturday, October 31, 2015 at 7:05:52 PM UTC, supe...@casperkitty.com wrote:
> On Thursday, October 15, 2015 at 8:10:23 PM UTC-5, Malcolm McLean wrote:
> > Generally it's more important to to portable and flexible than to
> > be as fast as possible. Sadly, C's FILE *s can't be used in all
> > situations.
> 
> I wonder what the performance implications would have been of making the
> first member of "file" be a pointer to a structure containing pointers to
> all the various functions a file could support. 
>
You could easily enough provide fgetc and/or fputc. You don't need any more
than that, and it makes custom streams easy to write.
But, sadly, indirect calls to function pointers held in passed-in
structures are relatively expensive on many systems, and you need
a lot of them if you're to read or write a whole file character by
character. So getc and putc were conventionally macros, that basically
resolved to an if statement, a buffer increment and read, and a 
conditional call to a subroutine to read the next buffer.

[toc] | [prev] | [next] | [standalone]

#74491

On Saturday, October 31, 2015 at 2:43:04 PM UTC-5, Malcolm McLean wrote:
> On Saturday, October 31, 2015 at 7:05:52 PM UTC, supercat wrote:
> > I wonder what the performance implications would have been of making the
> > first member of "file" be a pointer to a structure containing pointers to
> > all the various functions a file could support. 
> >
> You could easily enough provide fgetc and/or fputc. You don't need any more
> than that, and it makes custom streams easy to write.
> But, sadly, indirect calls to function pointers held in passed-in
> structures are relatively expensive on many systems, and you need
> a lot of them if you're to read or write a whole file character by
> character. So getc and putc were conventionally macros, that basically
> resolved to an if statement, a buffer increment and read, and a 
> conditional call to a subroutine to read the next buffer.

Performance would require at minimum multi-byte fread and fwrite, along
with fseek, ioctl, etc. and it would probably be helpful to have the
structure include start/end/current pointers for read/write buffers
and say that if a request can be satisfied using such pointers there is
no need to make the indirect function call, though I'm not sure how best
to distinguish the cases where a file needs to use a shared buffer pointer
for reading and writing from the case where it needs to use separate
buffers.

I/O devices which need to perform some action for each byte individually
could simply set themselves up without the buffers in question.

[toc] | [prev] | [next] | [standalone]

#74480

supercat@casperkitty.com writes:

> On Thursday, October 15, 2015 at 8:10:23 PM UTC-5, Malcolm McLean wrote:
>> Generally it's more important to to portable and flexible than to
>> be as fast as possible. Sadly, C's FILE *s can't be used in all
>> situations.
>
> I wonder what the performance implications would have been of making the
> first member of "file" be a pointer to a structure containing pointers to
> all the various functions a file could support.  That would have made it
> possible to pass many kinds of things to code which expects to read from
> a FILE*.

Just an aside...  BCPL has had some syntax to help do that for some time
though not in the original version.  I wonder if it was in the version
that was said to have influenced B and C.  The syntax looked like this:

   Read#(file, buf, len)
   
which, translated into C, it would mean

  (*file)[Read](file, buf, len)

'Read' would probably be (in C) an enumeration constant, but any
expression is permitted in that position.

And another aside linked to the talk about gcc's ({ ... }) extension to
permit blocks to appear in expressions.  BCPL did this is language with

  VALOF <statement>

being an expression whose value is determined by a RESULTIS <exp>
statement (a return in C terms).  The statement after VALOF is usually a
block, of course.

And then there were bit-field selectors.  I'm sure many people would
like to have had them in C.  Still, C is what it is.  Given the bad
press, it's surprising how popular it still is!

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#74492

On Saturday, October 31, 2015 at 2:55:10 PM UTC-5, Ben Bacarisse wrote:
> And then there were bit-field selectors.  I'm sure many people would
> like to have had them in C.  Still, C is what it is.  Given the bad
> press, it's surprising how popular it still is!

Tell me more.  What I've long wished for in C would have been a means of
indicating that ".someIdentifier" should be regarded as an alias for some
bits taken from other members of the structure.  That would have provided
all the utility of bitfields, but with a system-independent meaning.

[toc] | [prev] | [next] | [standalone]

#74500

supercat@casperkitty.com writes:

> On Saturday, October 31, 2015 at 2:55:10 PM UTC-5, Ben Bacarisse wrote:
>> And then there were bit-field selectors.  I'm sure many people would
>> like to have had them in C.  Still, C is what it is.  Given the bad
>> press, it's surprising how popular it still is!
>
> Tell me more.  What I've long wished for in C would have been a means of
> indicating that ".someIdentifier" should be regarded as an alias for some
> bits taken from other members of the structure.  That would have provided
> all the utility of bitfields, but with a system-independent meaning.

No, it's very different, though it might be applicable in some of the
same situations.  There is a construct

  SLCT len : shift : offset

which, at least conceptually, packs those three values into a word
(everything's a word in BCPL).  The OF operator takes a selector on the
left and evaluates its right hand side to get a pointer, p.  The OF
exporession denotes the 'len' bits 'shift' bits from the right-hand end
of the word at address p + offset.  The main thing of interest is that
OF expressions can be assigned to.

-- 
Ben.

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | comp.lang.c

csiph-web