Re: Working with files on different operating systems

From	"Morten W. Petersen" <morphex@gmail.com>
Newsgroups	comp.lang.c
Subject	Re: Working with files on different operating systems
Date	2015-10-17 04:01 +0200
Organization	Aioe.org NNTP Server
Message-ID	<mvsa60$au0$1@speranza.aioe.org> (permalink)
References	<mv7b7l$gru$1@speranza.aioe.org> <mv8ck5$5s9$1@dont-email.me> <mv8r8o$ru8$1@speranza.aioe.org> <45fdb118-722c-4c06-97c2-e79bad82dc63@googlegroups.com>

Show all headers | View raw

On 16.10.2015 03:09, Malcolm McLean wrote:
> On Friday, October 9, 2015 at 5:50:47 PM UTC+1, Morten W. Petersen wrote:
>>
>> I like the idea of searching for a string using for example strstr as
>> I assume that is about as fast as a search can get.
>>
> Yes and no.
> If you're doing lots of searches over a stable string (e.g. making
> an encyclopedia searchable), the fastest option is a suffix tree.
> But you've got to build the suffix tree, so it's unlikely that
> you can beat strstr() for temporary strings representing XML
> as it is loaded in.

OK, about as fast it can get without using special structures and
algorithms. :)

>> However, that the program can read from any stream and has as little
>> platform-specific code, are two significant good things..  I think
>> it is better to go back later and add optimizations.
>>
> Generally it's more important to to portable and flexible than to
> be as fast as possible. Sadly, C's FILE *s can't be used in all
> situations. Really streams are one of those things which need
> object-based design, a stream that is attached to an unreliable
> connection needs to be able to query the user for timeout,
> for example, sometimes the user will quit, other times he'll
> tell it to keep trying and go for a cup of coffee. An XML parser
> shouldn't be concerned with that.

Well, the parser can't do any parsing if it runs out of characters
to work on.  I think threading here can be one of the things that
makes working with streams more practical.

As for design without threads, maybe it is a good idea to allocate
some xml_item structs, compress data and so on while idling for input.

But you raise an interesting point, that the higher-level code or a
user might want to abort the parsing, and that's something that has
to be taken into account as well.

>> Same goes for the storage type; incremental reads will make the program
>> use less memory at peaks which counts towards stability and the ability
>> to deal with large files.
>>
>> The "window" will most of the time not be that large, but that doesn't
>> help if there are huge windows in a given XML stream.  And that's
>> something that has to be taken into account.
>>
> You can't design for everything.
> Is the goal to have really good performance on normal XML, even at
> the cost of poor performance or even failure on very abnormal
> documents? Or do you want to be able to handle absolutely anything
> that is within the capacity of the machine, reasonably quickly,
> even if that means the parser is quite a bit slower than competitors
> on normal XML?

Take for example an XML element attribute name.  It will probably not
be that long in most cases.  But the attribute value could be long,
very long or extremely long.

This content has to be allocated space for, parsed and so on, and if
one starts with a default of maybe 32 Unicode characters and reallocs
that down or takes the size * 1.5 for every increase in storage space,
isn't that an OK way to do it?

I'm prototyping now the print_tree function so that it uses recursion,
but everything that can recurse many times will be made into loops
instead, to avoid stack smashing, as well as giving useable feedback to
the higher level code as to what's wrong.

-Morten

Thread

Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 05:11 +0200
  Re: Working with files on different operating systems Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2015-10-08 22:12 -0600
  Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 00:18 -0500
    Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-09 11:13 +0100
      Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 18:05 +0200
        Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 12:49 -0500
        Re: Working with files on different operating systems Eric Sosman <esosman@comcast-dot-net.invalid> - 2015-10-09 17:32 -0400
        Re: Working with files on different operating systems Ian Collins <ian-news@hotmail.com> - 2015-10-10 12:37 +1300
      Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 12:48 -0500
        Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-09 19:53 +0100
          Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 21:09 +0200
            Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-09 20:28 +0100
              Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 21:40 +0200
                Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-09 15:59 -0400
                Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 22:07 +0200
                Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-09 16:24 -0400
                Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-10 03:08 +0200
                Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-09 21:50 +0100
        Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-09 14:54 -0400
          Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 21:05 +0200
            Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 21:47 +0200
            Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-09 15:53 -0400
              Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 22:01 +0200
                Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-09 16:05 -0400
          Re: Working with files on different operating systems glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-10-10 01:16 +0000
            Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-10 03:46 +0200
            Re: Working with files on different operating systems Eric Sosman <esosman@comcast-dot-net.invalid> - 2015-10-10 08:08 -0400
          Re: Working with files on different operating systems raltbos@xs4all.nl (Richard Bos) - 2015-10-10 17:02 +0000
        Re: Working with files on different operating systems Eric Sosman <esosman@comcast-dot-net.invalid> - 2015-10-09 15:53 -0400
    Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 18:00 +0200
      Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 12:59 -0500
        Re: Working with files on different operating systems Keith Thompson <kst-u@mib.org> - 2015-10-09 12:37 -0700
          Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 17:36 -0500
            Re: Working with files on different operating systems Keith Thompson <kst-u@mib.org> - 2015-10-09 16:52 -0700
              Re: Working with files on different operating systems raltbos@xs4all.nl (Richard Bos) - 2015-10-14 11:27 +0000
                Re: Working with files on different operating systems James Kuyper <jameskuyper@verizon.net> - 2015-10-14 11:27 -0400
                Re: Working with files on different operating systems Jorgen Grahn <grahn+nntp@snipabacken.se> - 2015-10-14 16:46 +0000
                Re: Working with files on different operating systems Keith Thompson <kst-u@mib.org> - 2015-10-14 10:18 -0700
              Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-14 23:06 -0500
    Re: Working with files on different operating systems Keith Thompson <kst-u@mib.org> - 2015-10-09 09:16 -0700
      Re: Working with files on different operating systems Les Cargill <lcargill99@comcast.com> - 2015-10-09 17:43 -0500
        Re: Working with files on different operating systems Keith Thompson <kst-u@mib.org> - 2015-10-09 16:57 -0700
          Re: Working with files on different operating systems Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-10-09 22:41 -0700
    Re: Working with files on different operating systems David Thompson <dave.thompson2@verizon.net> - 2015-10-31 06:50 -0400
  Re: Working with files on different operating systems Siri Cruz <chine.bleu@yahoo.com> - 2015-10-08 23:25 -0700
    Re: Working with files on different operating systems Geoff <geoff@invalid.invalid> - 2015-10-10 10:35 -0700
  Re: Working with files on different operating systems Eric Sosman <esosman@comcast-dot-net.invalid> - 2015-10-09 08:42 -0400
    Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-09 18:51 +0200
      Re: Working with files on different operating systems Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-10-15 18:09 -0700
        Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-17 04:01 +0200
          Re: Working with files on different operating systems Ian Collins <ian-news@hotmail.com> - 2015-10-17 15:55 +1300
            Re: Working with files on different operating systems Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-10-16 20:09 -0700
              Re: Working with files on different operating systems Ian Collins <ian-news@hotmail.com> - 2015-10-17 16:36 +1300
            Re: Working with files on different operating systems "Morten W. Petersen" <morphex@gmail.com> - 2015-10-17 05:10 +0200
        Re: Working with files on different operating systems supercat@casperkitty.com - 2015-10-31 12:05 -0700
          Re: Working with files on different operating systems Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-10-31 12:42 -0700
            Re: Working with files on different operating systems supercat@casperkitty.com - 2015-10-31 13:40 -0700
          Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-31 19:55 +0000
            Re: Working with files on different operating systems supercat@casperkitty.com - 2015-10-31 13:43 -0700
              Re: Working with files on different operating systems Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-31 21:38 +0000

csiph-web