Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #1119 > unrolled thread

find library in C

Back to article view | Back to comp.unix.programmer

Contents

  find library in C ZarkXe <zarkxe@gmail.com> - 2011-08-11 22:35 +0200
    Re: find library in C "Heinrich Wolf" <invalid@invalid.invalid> - 2011-08-11 23:16 +0200
      Re: find library in C Rui Maciel <rui.maciel@gmail.com> - 2011-08-12 15:50 +0100
        Re: find library in C Rui Maciel <rui.maciel@gmail.com> - 2011-08-12 16:20 +0100
    Re: find library in C Rui Maciel <rui.maciel@gmail.com> - 2011-08-12 15:29 +0100
      Re: find library in C "Heinrich Wolf" <invalid@invalid.invalid> - 2011-08-12 17:03 +0200
        Re: find library in C "Heinrich Wolf" <invalid@invalid.invalid> - 2011-08-12 17:09 +0200
        Re: find library in C Rui Maciel <rui.maciel@gmail.com> - 2011-08-12 16:31 +0100
      Re: find library in C Nicolas George <nicolas$george@salle-s.org> - 2011-08-12 16:24 +0000
        Re: find library in C Rui Maciel <rui.maciel@gmail.com> - 2011-08-12 18:58 +0100
          Re: find library in C Nicolas George <nicolas$george@salle-s.org> - 2011-08-12 19:16 +0000
            Re: find library in C Rui Maciel <rui.maciel@gmail.com> - 2011-08-13 13:42 +0100
              Re: find library in C Nicolas George <nicolas$george@salle-s.org> - 2011-08-13 15:06 +0000
          Re: find library in C Ian Collins <ian-news@hotmail.com> - 2011-08-13 11:40 +1200
            Re: find library in C Rui Maciel <rui.maciel@gmail.com> - 2011-08-13 11:08 +0100
        Re: find library in C William Ahern <william@wilbur.25thandClement.com> - 2011-08-12 12:58 -0700
          Re: find library in C Nicolas George <nicolas$george@salle-s.org> - 2011-08-12 20:31 +0000
            Re: find library in C William Ahern <william@wilbur.25thandClement.com> - 2011-08-12 14:47 -0700
            Re: find library in C Rui Maciel <rui.maciel@gmail.com> - 2011-08-13 14:30 +0100
              Re: find library in C William Ahern <william@wilbur.25thandClement.com> - 2011-08-13 18:54 -0700
      Re: find library in C Rainer Weikusat <rweikusat@mssgmbh.com> - 2011-08-12 18:55 +0100
        Re: find library in C Rui Maciel <rui.maciel@gmail.com> - 2011-08-12 19:37 +0100
          Re: find library in C Rainer Weikusat <rweikusat@mssgmbh.com> - 2011-08-12 20:21 +0100
            Re: find library in C Rui Maciel <rui.maciel@gmail.com> - 2011-08-13 10:23 +0100
              Re: find library in C Rainer Weikusat <rweikusat@mssgmbh.com> - 2011-08-13 14:53 +0100
    Re: find library in C "Heinrich Wolf" <invalid@invalid.invalid> - 2011-08-15 13:58 +0200

Page 1 of 2  [1] 2  Next page →

#1119 — find library in C

Hello all,

I find a library or function in C for create/read/write/modify files 
configuration (*.conf or *.ini)/

Thanks in advance,
ZarkXe

[toc] | [next] | [standalone]

#1120

"ZarkXe" <zarkxe@gmail.com> schrieb im Newsbeitrag 
news:j21e88$fba$1@speranza.aioe.org...
> Hello all,
>
> I find a library or function in C for create/read/write/modify files 
> configuration (*.conf or *.ini)/
>
> Thanks in advance,
> ZarkXe

Google is your friend!
http://code.google.com/p/inih/
http://www.jbox.dk/sanos/source/
http://ndevilla.free.fr/iniparser/
and lots of more 

[toc] | [prev] | [next] | [standalone]

#1124

Heinrich Wolf wrote:

> Google is your friend!
> http://code.google.com/p/inih/
> http://www.jbox.dk/sanos/source/
> http://ndevilla.free.fr/iniparser/
> and lots of more

In some cases it isn't.  For example, the first item on your list consists 
of a nasty, ugly brute-force hack which, although it may work, it leaves the 
developer worse off than if he just developed himself a similar brute-force 
routine.

Meanwhile, a while ago I've developed a small C++ library which parses an 
ini-type language and builds a nested std::map which maps [section][label] 
to [values], and supports comments.  The code is available at:

http://minip.git.sourceforge.net/git/gitweb.cgi?p=minip/minip;a=tree;h=9cec5d576c8ab925454972256d2402e489377a6b;hb=9cec5d576c8ab925454972256d2402e489377a6b


I haven't released a package yet, mainly because I've only wrote it to test 
a couple of generic classes which I use as a basis to my LL parsers.  


Hope this helps,
Rui Maciel

[toc] | [prev] | [next] | [standalone]

#1127

Rui Maciel wrote:

> I haven't released a package yet, mainly because I've only wrote it to
> test a couple of generic classes which I use as a basis to my LL parsers.

I've created a small package for this small piece of code.  It's available 
at:

https://sourceforge.net/projects/minip/files/minip/v0.2/minip-0.2.tar.gz/download



Rui Maciel

[toc] | [prev] | [next] | [standalone]

#1123

ZarkXe wrote:

> Hello all,
> 
> I find a library or function in C for create/read/write/modify files
> configuration (*.conf or *.ini)/

This sort of library isn't very common because it's functionality is trivial 
to implement, and it tends to be tied to what the developer intends to 
achieve.  In C, at least when compared to C++, I believe the need to develop 
your own custom library is even more pressing, as the language doesn't 
provide a set of standard data structures nor supports generic programming.

So, again, you are better off developing your own.  Once you do it, 
considering that the INI-type language is very simple, you will see that it 
is terribly easy to do.


Rui Maciel

[toc] | [prev] | [next] | [standalone]

#1125

"Rui Maciel" <rui.maciel@gmail.com> schrieb im Newsbeitrag 
news:4e453be5$0$31423$a729d347@news.telepac.pt...
...
> So, again, you are better off developing your own.  Once you do it,
> considering that the INI-type language is very simple, you will see that 
> it
> is terribly easy to do.

and I think that is what many programmers do, because I have seen many *.ini 
files which do not conform to the Microsoft implementation of reading and 
writing an ini file. In fact I myself have designed a "string.ini" which is 
not accessed by GetPrivateProfileString() and reads like this:

S[0,0]=_SpracheX
S[0,1]=Deutsch
S[0,2]=Française
S[0,4]=Italiano
S[0,5]=Norsk
S[0,6]=Español
S[0,7]=Nederlands
S[0,7]=Nederlands
S[0,8]=ÅëëçíéêÜ
S[0,9]=Türkçe
S[0,10]=Magyar
S[0,11]=
S[0,12]=
S[0,13]=
S[0,14]=Português
S[0,15]=
S[0,16]=

S[1,0]=_OK
S[1,1]=OK
S[1,2]=OK
S[1,3]=OK
S[1,2]=OK
S[1,3]=OK
S[1,4]=OK
S[1,5]=OK
S[1,6]=Si

...

I also have seen e.g. Unify Team developers's sql.ini , which may read like 
this:

[infogtwy]
remotedbname=productive_db, DATABASE 
db, -hproductive_host -ssqlexec -vproductive_server
remotedbname=test_db, DATABASE db, -htest_host -ssqlexec -vtest_server
defconnect=off
...

Note, that it contains 2 different and both valid configurations on both the 
same key

[toc] | [prev] | [next] | [standalone]

#1126

"Heinrich Wolf" <invalid@invalid.invalid> schrieb im Newsbeitrag 
news:j23fc0$knl$1@news.m-online.net...
...
> [infogtwy]
> remotedbname=productive_db, DATABASE 
> db, -hproductive_host -ssqlexec -vproductive_server
> remotedbname=test_db, DATABASE db, -htest_host -ssqlexec -vtest_server
> defconnect=off
> ...

Sorry! My outlook express inserted line breaks.

I write it again with indicating the start of a line by **
**[infogtwy]
**remotedbname=productive_db, DATABASE 
db, -hproductive_host -ssqlexec -vproductive_server
**remotedbname=test_db, DATABASE db, -htest_host -ssqlexec -vtest_server
**defconnect=off

[toc] | [prev] | [next] | [standalone]

#1128

Heinrich Wolf wrote:

> 
> "Rui Maciel" <rui.maciel@gmail.com> schrieb im Newsbeitrag
> news:4e453be5$0$31423$a729d347@news.telepac.pt...
> ...
>> So, again, you are better off developing your own.  Once you do it,
>> considering that the INI-type language is very simple, you will see that
>> it
>> is terribly easy to do.
> 
> and I think that is what many programmers do, because I have seen many
> *.ini files which do not conform to the Microsoft implementation of
> reading and writing an ini file. In fact I myself have designed a
> "string.ini" which is not accessed by GetPrivateProfileString() and reads
> like this:
> 
> S[0,0]=_SpracheX
> S[0,1]=Deutsch
> S[0,2]=Française
> S[0,4]=Italiano
> S[0,5]=Norsk
> S[0,6]=Español
> S[0,7]=Nederlands
> S[0,7]=Nederlands
> S[0,8]=ÅëëçíéêÜ
> S[0,9]=Türkçe
> S[0,10]=Magyar
> S[0,11]=
> S[0,12]=
> S[0,13]=
> S[0,14]=Português
> S[0,15]=
> S[0,16]=
> 
> S[1,0]=_OK
> S[1,1]=OK
> S[1,2]=OK
> S[1,3]=OK
> S[1,2]=OK
> S[1,3]=OK
> S[1,4]=OK
> S[1,5]=OK
> S[1,6]=Si
> 
> ...
> 
> I also have seen e.g. Unify Team developers's sql.ini , which may read
> like this:
> 
> [infogtwy]
> remotedbname=productive_db, DATABASE
> db, -hproductive_host -ssqlexec -vproductive_server
> remotedbname=test_db, DATABASE db, -htest_host -ssqlexec -vtest_server
> defconnect=off
> ...
> 
> Note, that it contains 2 different and both valid configurations on both
> the same key


That's very true.  In fact, these small config files are intended to scratch 
a particular itch, and naturally those who have that itch know best how to 
scratch it.

The ini parser which I've recently released[1] supports a language which 
differs a bit from the typical ini format, which is:

<example>

[sectionA]	; this is a comment 
"option1" = "foo"
"option2" = "bar"  
; commenting yet again
"option3" = "maybe" ; can comment here, too
; the following options will override the values which have been previously 
set
"option1" = "true" 
"option2" = "false"
"option3" = "true"

[sectionB]	; yet another section
"option1" = "foo"
"option2" = "bar"  
; you get the picture...

</example>


The only reason why I've developed this language like this is that it is 
considerably simpler to parse text strings which are enclosed in quotation 
marks.  Meanwhile, I use a different ini-type language in my projects, which 
is a bit more robust than this.

So, the moral of the story is that, for such a simple task, it is far better 
to just roll our own code.  Our very own inadequate solution is better than 
someone else's inadequate solution.


Rui Maciel


[1] 
https://sourceforge.net/projects/minip/files/minip/v0.2/minip-0.2.tar.gz/download

[toc] | [prev] | [next] | [standalone]

#1129

Rui Maciel , dans le message
<4e453be5$0$31423$a729d347@news.telepac.pt>, a écrit :
> This sort of library isn't very common because it's functionality is trivial 
> to implement, and it tends to be tied to what the developer intends to 
> achieve.

It seems trivial, but it is not at all, if you want to do things properly.
If you write your own library, you have to document the answer to the
following questions:

- How do I quote strings?

- How do I include the quotation mark inside a string?

- Are the strings sequences of bytes or of characters? And if characters, in
  what encoding?

- Can I split a long string on several lines?

- How do I insert a comment?

- If I comment out a multi-line string, does the comment span?

- How can I include a control character in a string?

- etc.

Usually, with a solution designed by someone who thought it was trivial, the
answer to a lot of these questions will be "I do not know" or "you can't".

Using an existing library for a well-known syntax, such as XML or JSON, has
the double advantage that (1) the library will usually take care of allowing
corner cases, such as including the quotation character inside a string, and
(2) the users will usually be already be familiar with the syntax.

[toc] | [prev] | [next] | [standalone]

#1131

Nicolas George wrote:

> Rui Maciel , dans le message
> <4e453be5$0$31423$a729d347@news.telepac.pt>, a écrit :
>> This sort of library isn't very common because it's functionality is
>> trivial to implement, and it tends to be tied to what the developer
>> intends to achieve.
> 
> It seems trivial, but it is not at all, if you want to do things properly.
> If you write your own library, you have to document the answer to the
> following questions:
> 
> - How do I quote strings?
> 
> - How do I include the quotation mark inside a string?
> 
> - Are the strings sequences of bytes or of characters? And if characters,
> in
>   what encoding?
> 
> - Can I split a long string on several lines?
> 
> - How do I insert a comment?
> 
> - If I comment out a multi-line string, does the comment span?
> 
> - How can I include a control character in a string?
> 
> - etc.

In other words, you need to define the language.  This is trivial.


> Usually, with a solution designed by someone who thought it was trivial,
> the answer to a lot of these questions will be "I do not know" or "you
> can't".

...which can just as easily be followed by checking the answer for those 
questions and/or implement the support for the features which are really 
needed.  For example, the library which I've provided that supports an ini-
type language[1] doesn't support basic features such as escaped characters 
in the strings and value types other than text strings.  Yet, it would be 
trivial to add support for those features, although the data structure would 
need to be refurbished.

Meanwhile, when people tend to just swallow ready-made solutions such as 
generic XML and JSON parsers, they tend to needlessly complicate their lives 
without gaining any added value.  In fact, they tend to inadvertedly make 
their job even harder than it would be if they simply rolled a parser for 
their own language.  After all, when you adopt a third-party parser you are 
effectively replacing the task of developing a parser that builds your data 
structure from a byte stream with two separate tasks: implementing a third-
party library that converts a byte stream to a tree structure and then 
develop a crude parser to parse that tree structure in order to build your 
data structure.  So, you don't avoid any work, nor the need to develop a 
parser, and you needlessly bloat your code, without gaining anything.


> Using an existing library for a well-known syntax, such as XML or JSON,
> has the double advantage that (1) the library will usually take care of
> allowing corner cases, such as including the quotation character inside a
> string, and (2) the users will usually be already be familiar with the
> syntax.

These details are irrelevant.  There are a considerable number of 
applications where a developer only needs a basic map between a key and a 
value, where the value doesn't even need to be of any specific type other 
than a basic text string.  

Adding to this, when you employ languages such as XML or JSON to exchange 
information, what you are effectively doing is defining a language which is 
a proper subset of those languages.  A language which is a subset of any 
other language is still a language on it's own.  This means that you can 
still employ XML or JSON in your day-to-day usage but you are obviously 
better off developing your own parser for your specific language, due to 
factors such as simplicity, efficiency improvements and the ability to 
perform any sanity check right in the first parsing.


Rui Maciel

[1] 
https://sourceforge.net/projects/minip/files/minip/v0.2/minip-0.2.tar.gz/download

[toc] | [prev] | [next] | [standalone]

#1133

Rui Maciel , dans le message
<4e456ce8$0$31420$a729d347@news.telepac.pt>, a écrit :
> In other words, you need to define the language.  This is trivial.

Depends on what you call trivial. Multiplying two 50×50 integer matrices is
also "trivial", but if you try to do it with pen and paper you will waste a
lot of time and certainly make a lot of errors.

Writing a parser is just the same: it is essentially trivial, but it takes
time and the result has usually errors.

>	   For example, the library which I've provided that supports an ini-
> type language[1] doesn't support basic features such as escaped characters 
> in the strings

In other words, any application based on this library would have very
annoying limitations on what can be written in the config file. This is
called reinventing a square wheel, and it just proves my point.

> Meanwhile, when people tend to just swallow ready-made solutions such as 
> generic XML and JSON parsers, they tend to needlessly complicate their lives 
> without gaining any added value.  In fact, they tend to inadvertedly make 
> their job even harder than it would be if they simply rolled a parser for 
> their own language.

This is just a gratuitous affirmation backed by no evidence.

>			After all, when you adopt a third-party parser you are 
> effectively replacing the task of developing a parser that builds your data 
> structure from a byte stream with two separate tasks: implementing a third-
> party library that converts a byte stream to a tree structure and then 
> develop a crude parser to parse that tree structure in order to build your 
> data structure.

That is true, but parsing and validating are two different tasks, and mixing
them together usually only makes maintaining the software harder.

(As a side note, with XML, a large part of the validation could be done
using DTDs or schemas. But I would not advocate the use of these features,
since they are complex techniques, both for the implementer (validating
parsers are larger and more bogus than simple parsers) and the user.)

>		   So, you don't avoid any work, nor the need to develop a 
> parser, and you needlessly bloat your code, without gaining anything.

That is just not true: all the work of parsing (as opposed to validating:
handling escaped characters, skipping whitespace and comments, etc.) has
been spared.

> These details are irrelevant.

This is not irrelevant at all. As an user with experience, I much prefer
reading "the config file is XML" rather than "the config file is made of
sections blah blah comments start with blah blah strings with special
characters must be quoted blah blah": three characters versus a whole
paragraph.

>				There are a considerable number of 
> applications where a developer only needs a basic map between a key and a 
> value, where the value doesn't even need to be of any specific type other 
> than a basic text string.  

Obviously, generic parsers are able to parse such simple structures.

And applications have a tendency to grow and end up needing more complex
configuration, be it only strings with embedded quotes or newlines.

> Adding to this, when you employ languages such as XML or JSON to exchange 
> information, what you are effectively doing is defining a language which is 
> a proper subset of those languages.

Indeed.

>				       A language which is a subset of any 
> other language is still a language on it's own.

Also true.

>						   This means that you can 
> still employ XML or JSON in your day-to-day usage but you are obviously 
> better off developing your own parser for your specific language, due to 
> factors such as simplicity, efficiency improvements and the ability to 
> perform any sanity check right in the first parsing.

Not obvious at all, and in fact mostly wrong.

Simplicity: wrong, see above.

Efficiency: I grant you this one, generic parsers tend to do a lot of
mallocs/strdup that could be avoided with a more direct approach. But
efficiency is usually not a concern when parsing a config file.

Improvements: the generic parsers are already capable of handling complex,
recursive structures, there is no additional work when (not "if") the need
arises.

Validation in the first pass: usually a bad design in the first place.

[toc] | [prev] | [next] | [standalone]

#1142

Nicolas George wrote:

> Rui Maciel , dans le message
> <4e456ce8$0$31420$a729d347@news.telepac.pt>, a écrit :
>> In other words, you need to define the language.  This is trivial.
> 
> Depends on what you call trivial. Multiplying two 50×50 integer matrices
> is also "trivial", but if you try to do it with pen and paper you will
> waste a lot of time and certainly make a lot of errors.

I don't understand your example.  I happen to deal with matrix algebra in a 
day-to-day basis and the last time I needed to do this sort of calculation 
by hand it was way back in a linear algebra class I took, and it didn't go 
beyond simple 5*5 matrices.  This is due to the fact that, when dealing with 
algorithms, if you understand how they work and if you understand how they 
scale then you are able to guarantee that your implemenation does work if 
you guarantee that it works in a set of simple test cases and you guarantee 
that it doesn't break down when it scales.  This holds true for matrix 
algebra and parsers.


> Writing a parser is just the same: it is essentially trivial, but it takes
> time and the result has usually errors.

Yes, it's possible that we inadvertedly add some bugs to our 
implementations.  Yet, if we develop our routines well enough and if we test 
them adequately, it is possible to iron out the bugs in no time.  For 
example, last week I've developed an LL parser for a JSON-based language 
whose parser table is about 50*140.  When I finished implementing it I've 
had to deal with a number of bugs, mainly due to having placed the wrong 
terminal token in some rules.  Nevertheless, after performing a series of 
tests, I was able to fix all the bugs in about 20 minutes. Granted, this by 
no way means that this parser is free from any bug whatsoever, but I did get 
a fully working parser that works in every conceivable (or, better yet, 
conceived) scenario.

In contrast, I don't believe it would be easier to achieve this by employing 
a generic parser and then parsing the document structure it generates.



>>For example, the library which I've provided that supports an ini-
>> type language[1] doesn't support basic features such as escaped
>> characters in the strings
> 
> In other words, any application based on this library would have very
> annoying limitations on what can be written in the config file. This is
> called reinventing a square wheel, and it just proves my point.

Yes, it does have limitations which can be euphemistically described as 
being annoying.  Yet, this was a small test parser which I've developed in 
about half an hour, just to be able to test an idea on how to develop a 
generic base class for my parsers.  I never intended to rely on it for any 
of my applications, nor do I plan on do so in the future.  In fact, the code 
I do employ to handle config files does support a ini-type language which is 
quite more capable and is much more expressive, that includes support for 
escaped characters, data types and lists of those data types.  This parser 
didn't took much more work to do than this, because it is in fact that easy 
to do it.  Just to give you an idea, in order to add support for numerical 
data types in the library that I provied[1] I only need to add a single rule 
to recognize a numerical terminal token and a couple of production rules for 
the language to accept it.  It's that easy.


>> Meanwhile, when people tend to just swallow ready-made solutions such as
>> generic XML and JSON parsers, they tend to needlessly complicate their
>> lives
>> without gaining any added value.  In fact, they tend to inadvertedly make
>> their job even harder than it would be if they simply rolled a parser for
>> their own language.
> 
> This is just a gratuitous affirmation backed by no evidence.

Have you ever tried it?  When you do you I guarantee that you will 
understand and acknowledge this as blatantly obvious.

One of the sad, degrading consequences of the mindless overreliance on 
marketing buzzwords such as XML is that due to this developers tend to 
completely lose perspective of what they are actually trying to do and how 
they are doing it.  A culture has been formed where a developer, when faced 
with the need to import data or implement a communications protocol, tends 
not to actually think through his problem.  Instead, the developer 
mindlessly jumps to the assumption that what he needs to do is employ XML, 
whether that makes sense or not.  From there he tries to manipulate and 
distort his task in order to shoe-horn XML into what he needed to do, 
whether it makes sense or not.  With this comes the absurd assumption that 
if the developer needs to handle XML then he is required to employ a generic 
XML parser, which he then is forced to learn how to use and control it's 
idiossincracies, all this before even being in place to actually extract 
data from any stream.

This, in short, means one thing: these developers don't know how to import 
data.  They don't know how to read a stream/file, or even why this is even 
needed, they don't know how to parse a stream of data, they don't know how 
to validate a document structure and in some cases they don't even know how 
ot write/export their data.  All they know is XML, and by XML it means they 
only know how to install a third-party library, pass a set of parameters to 
that library and proceed with a series of arcane DOM-like voodoo on a data 
structure with hopes that they can adequately make it work.  Meanwhile they 
lost any perspective on what they tried to do, and how hard they made their 
task out to be.


>>After all, when you adopt a third-party parser you are
>> effectively replacing the task of developing a parser that builds your
>> data structure from a byte stream with two separate tasks: implementing a
>> third- party library that converts a byte stream to a tree structure and
>> then develop a crude parser to parse that tree structure in order to
>> build your data structure.
> 
> That is true, but parsing and validating are two different tasks, and
> mixing them together usually only makes maintaining the software harder.

Not quite.  When a program validates a given XML document, what that program 
does is perform a series of checks to see if that document complies with the 
rules that make XML in fact XML.  If a DTD and equivalent definition is 
provided then that series of checks is extended to check if a given document 
conforms with the definition of a language which is a subset of XML.  This 
means that all a validator does is perform the same task that a parser 
performs, no more no less.


> (As a side note, with XML, a large part of the validation could be done
> using DTDs or schemas. But I would not advocate the use of these features,
> since they are complex techniques, both for the implementer (validating
> parsers are larger and more bogus than simple parsers) and the user.)
> 
>>So, you don't avoid any work, nor the need to develop a
>> parser, and you needlessly bloat your code, without gaining anything.
> 
> That is just not true: all the work of parsing (as opposed to validating:
> handling escaped characters, skipping whitespace and comments, etc.) has
> been spared.

You don't know what you are talking about.  The parser which I've 
released[1], which is extremely simple and which was developed without 
putting any serious effort into it, skips whitespaces and comments quite 
effortlessly, all this while guaranteeing that the documents comply with the 
language definition.  If I intended to add support for escaped characters 
then it would be quite possible to do so.  This sort of stuff is terribly 
trivial to implement if you develop your own parser.


>> These details are irrelevant.
> 
> This is not irrelevant at all. As an user with experience, I much prefer
> reading "the config file is XML" rather than "the config file is made of
> sections blah blah comments start with blah blah strings with special
> characters must be quoted blah blah": three characters versus a whole
> paragraph.

Your comment doesn't make sense.  Every document that you ever handled 
consists of a language.  Even XML-based document formats are languages, 
which happen to be a strict subset of XML.  Whenever you need to handle such 
a file, you need to comply with that language's definition.  For example, if 
you need to tweak a config file which consists of a subset of XML then you 
need to know what to tweak and how to tweak it in order to get the result 
you seek.  You may not need to memorize the entire language definition, but 
you do have to know that element X relates to option B, and that you can't 
tweak option B by adding/editing element Y.

Another reason why your comment doesn't make sense is that your ability to 
read XML config files doesn't depend on what technique has been employed in 
a program to extract information from that file.  Whether a program employs 
a generic parser with a validator and DOM voodoo or whether a program 
employs a custom parser, the language is always the same.


>>There are a considerable number of
>> applications where a developer only needs a basic map between a key and a
>> value, where the value doesn't even need to be of any specific type other
>> than a basic text string.
> 
> Obviously, generic parsers are able to parse such simple structures.
> 
> And applications have a tendency to grow and end up needing more complex
> configuration, be it only strings with embedded quotes or newlines.

The thing is, if someone only requires such a simple document format then 
using/writing a small parser such as the one I've presented here[1] does 
exactly what they need.  In contrast, even with such as simple language, if 
you need to employ a generic parser then you are forced to adopt a bloated 
generic third party parser, enforce your language by employing some form of 
validator and employ DOM-type voodoo to extract your information from a 
file.  This represents the difference between a lean 600-LoC component that 
you've developed yourself and a massive, bloated dependency which you don't 
even understand what it does under the hood.

Then, if the requirements change and suddenly the document format becomes a 
bit more complex then the simple custom parser does require tweaking to 
support those changes.  And the thing is, so does the approach which relies 
on a generic third-party parser, validator and DOM-type voodoo.  So, exactly 
what is there to be gained?



>>This means that you can
>> still employ XML or JSON in your day-to-day usage but you are obviously
>> better off developing your own parser for your specific language, due to
>> factors such as simplicity, efficiency improvements and the ability to
>> perform any sanity check right in the first parsing.
> 
> Not obvious at all, and in fact mostly wrong.
> 
> Simplicity: wrong, see above.

Nowhere you've came close to assert that employing a generic parser, 
validator and DOM-voodoo does represent a simpler approach.  


> Efficiency: I grant you this one, generic parsers tend to do a lot of
> mallocs/strdup that could be avoided with a more direct approach. But
> efficiency is usually not a concern when parsing a config file.
> 
> Improvements: the generic parsers are already capable of handling complex,
> recursive structures, there is no additional work when (not "if") the need
> arises.

This is absurd.  You are able to support infinitely recursive structures 
with a single production rule.  For example, here is a production for a 
JSON-based ini-type language:

<example>
<start>	= <document> <EOF>
<document>	= <object>

<object>	=  '{' '}'
	=/ '{' <label:value> *(',' <label:value>]) '}'

<value>	= <text string>

<label:value> = <text string> ':' <value>
</example>


And now, here is a production for a JSON-based language which supports 
infinite recursion:

<example>
<start>	= <document> <EOF>
<document>	= <object>

<object>	=  '{' '}'
	=/ '{' <label:value> *(',' <label:value>]) '}'

<value>	= <text string>
	=/ <object>

<label:value> = <text string> ':' <value>
</example>


How is this complicated?  It represents a single production rule.  For a 
comparisson, can you provide a schema for JSON that validates this language?


> Validation in the first pass: usually a bad design in the first place.

You simply don't know what you are talking about.


Rui Maciel


[1] 
https://sourceforge.net/projects/minip/files/minip/v0.2/minip-0.2.tar.gz/download

[toc] | [prev] | [next] | [standalone]

#1145

Rui Maciel , dans le message
<4e467455$0$31418$a729d347@news.telepac.pt>, a écrit :
> I don't understand your example.

My example is just there to illustrate that what you call "trivial" can
still require time and effort, and end up wrong.

> In contrast, I don't believe it would be easier to achieve this by employing 
> a generic parser and then parsing the document structure it generates.

And I believe it would.

>				      I never intended to rely on it for any 
> of my applications, nor do I plan on do so in the future.

Which raises the question of why you bothered to publish it...

> Have you ever tried it?

Yes, several times. And I also tried the other approaches several times.

> One of the sad, degrading consequences of the mindless overreliance on 
> marketing buzzwords such as XML is that due to this developers tend to 
> completely lose perspective of what they are actually trying to do and how 
> they are doing it.

The fact that XML has been abused as a buzzword does not necessarily imply
that the thing itself has no technical merit.

> This, in short, means one thing: these developers don't know how to import 
> data.

Yes, they don't. And they wouldn't be able to write a parser either. This
does not prove anything.

Let me state it clearly: I believe that a good developers absolutely need
to be able to write a parser; not knowing how would be a gap in their
competence. But that does not mean they should always use that competence.

Knowing how to use an existing library, with experience on at least two
different ones, is also a necessary competence.

Only that way can they decide, for each particular situation, what solution
is the best.

> Not quite.  When a program validates a given XML document

That was not what I meant, you are answering completely beside the point.

> You don't know what you are talking about.  The parser which I've 
> released[1], which is extremely simple and which was developed without 
> putting any serious effort into it, skips whitespaces and comments quite 
> effortlessly, all this while guaranteeing that the documents comply with the 
> language definition.

This is not true. Your parser reads key-values pairs in sections, but that
is not the whole language definition. The language definition, in a real
case, also includes the list of valid keys, and the accepted values for
these keys. Your parser does not care about that.

In fact, your parser does exactly the same thing as a generic JSON or
(non-validating) XML parser. Except it does it badly: no escaping, no
documentation.

I will grant you one thing: as the generic language you recognize is less
generic, your parser can have a simpler API. But this is a two-edged
advantage, as it implies that if your needs come to exceed the possibility
of the generic language, you would have to rework completely the API.

>			If I intended to add support for escaped characters 
> then it would be quite possible to do so.  This sort of stuff is terribly 
> trivial to implement if you develop your own parser.

It's even simpler if you use existing libraries: it is already done.

> Your comment doesn't make sense.

It makes perfect sense, you just did not bother to try to understand it. Let
me give you an example:

Here is an example of config file I just invented:

contact dantes@pharaon.com     'Edmond Dantès'
contact mercedes@catalans.fr   'Mercédès'
contact mondego@catalans.fr    'Fernand Mondego'
contact avrigny@solicitors.com 'M. Avrigny'

Here is the same file using a XML syntax:

<?xml version='1.0'?>
<contacts>
  contact mail='dantes@pharaon.com'     name='Edmond Dantès' />
  contact mail='mercedes@catalans.fr'   name='Mercédès' />
  contact mail='mondego@catalans.fr'    name='Fernand Mondego' />
  contact mail='avrigny@solicitors.com' name='M. Avrigny' />
</contacts>

I grant you it is slightly more verbose, but not much and still quite
readable.

Now, exercise for the reader: the correct name of the solicitor is
d'Avrigny, not Avrigny, as he's a noble. How do you correct it in the config
file?

With the XML version, that's easy: you already know the generic syntax for
XML, and you know that you can use double quotes: name="M. d'Avrigny", or
use a character entity: name='M. d&apos;Avrigny'.

With the ad-hoc syntax, you need to peruse the documentation, with 10%
chances that is is not possible and 10% chances that it is possible but not
explained in the doc.


> Another reason why your comment doesn't make sense is that your ability to 
> read XML config files doesn't depend on what technique has been employed in 
> a program to extract information from that file.  Whether a program employs 
> a generic parser with a validator and DOM voodoo or whether a program 
> employs a custom parser, the language is always the same.

That is true, but that does not make my argument nonsensical.

The position I support is this: is is usually a better idea to use an
existing library for a well-known syntax.

If you read it carefully, you will notice it has two parts: existing
library, well-known syntax. This argument was supporting the latest part.

> The thing is, if someone only requires such a simple document format then 
> using/writing a small parser such as the one I've presented here[1] does 
> exactly what they need.

Not "exactly", see above.

>			   In contrast, even with such as simple language, if 
> you need to employ a generic parser then you are forced to adopt a bloated 

You are assuming that generic parsers are always bloated, that is a petitio
principii.

> generic third party parser, enforce your language by employing some form of 
> validator

If the required syntax is as simple as you have supposed, enforcing the
language is just a matter of looping on the child nodes to check they are
empty.

>	    and employ DOM-type voodoo to extract your information from a 
> file.

Again, just looping on the attributes names and values to get the required
values.

>	This represents the difference between a lean 600-LoC component that 
> you've developed yourself

600 LoC for that task, that's at least 500 too much

>			    and a massive, bloated dependency which you don't 

Petitio principii.

> even understand what it does under the hood.

And another petitio principii.

> Nowhere you've came close to assert that employing a generic parser, 
> validator and DOM-voodoo does represent a simpler approach.  

And that is called burning a straw man.

> This is absurd.  You are able to support infinitely recursive structures 
> with a single production rule.

As long as the language and its implementation were already designed to
allow it easily, yes, that's easy. But that's also cheating.

> https://sourceforge.net/projects/minip/files/minip/v0.2/minip-0.2.tar.gz/download

Is it supposed to be shown as an exemplary Unix code? Where is the enclosing
directory? Where is the makefile? Where is the licence?

And let us not forget the gaping security hole in IniParser::fill.

[toc] | [prev] | [next] | [standalone]

#1138

On 08/13/11 05:58 AM, Rui Maciel wrote:
> Nicolas George wrote:
>
>> Using an existing library for a well-known syntax, such as XML or JSON,
>> has the double advantage that (1) the library will usually take care of
>> allowing corner cases, such as including the quotation character inside a
>> string, and (2) the users will usually be already be familiar with the
>> syntax.
>
> These details are irrelevant.  There are a considerable number of
> applications where a developer only needs a basic map between a key and a
> value, where the value doesn't even need to be of any specific type other
> than a basic text string.

Which is precisely what JSON gives you, in a very concise form:

{"key":"value"}

Which is trivial to parse from scratch.  So by using an well known 
syntax, the programmer has the choice between a hand written parser, or 
an existing library.

> Adding to this, when you employ languages such as XML or JSON to exchange
> information, what you are effectively doing is defining a language which is
> a proper subset of those languages.  A language which is a subset of any
> other language is still a language on it's own.

Why is that a problem?  You can still use an existing library to do the 
paring and use the parser output as the input to your interpreter.

> This means that you can
> still employ XML or JSON in your day-to-day usage but you are obviously
> better off developing your own parser for your specific language, due to
> factors such as simplicity, efficiency improvements and the ability to
> perform any sanity check right in the first parsing.

Until your language subset grows. I generally use JSON for configuration 
files and by using a generic parser, I don't have to worry about 
extending the configuration syntax.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#1141

Ian Collins wrote:

> On 08/13/11 05:58 AM, Rui Maciel wrote:
>> Nicolas George wrote:
>>
>>> Using an existing library for a well-known syntax, such as XML or JSON,
>>> has the double advantage that (1) the library will usually take care of
>>> allowing corner cases, such as including the quotation character inside
>>> a string, and (2) the users will usually be already be familiar with the
>>> syntax.
>>
>> These details are irrelevant.  There are a considerable number of
>> applications where a developer only needs a basic map between a key and a
>> value, where the value doesn't even need to be of any specific type other
>> than a basic text string.
> 
> Which is precisely what JSON gives you, in a very concise form:
> 
> {"key":"value"}
> 
> Which is trivial to parse from scratch.  So by using an well known
> syntax, the programmer has the choice between a hand written parser, or
> an existing library.

Indeed.


>> Adding to this, when you employ languages such as XML or JSON to exchange
>> information, what you are effectively doing is defining a language which
>> is
>> a proper subset of those languages.  A language which is a subset of any
>> other language is still a language on it's own.
> 
> Why is that a problem?  You can still use an existing library to do the
> paring and use the parser output as the input to your interpreter.

This isn't a problem.  In fact, it is one of the main points (if not the 
whole point) of adopting languages such as XML and JSON as the basis for 
domain-specific languages.  Yet, it is necessary to point out that when 
someone employs XML or JSON as a basis for their languages, a valid XML/JSON 
document may not be a valid document in their custom language.  This means 
that the custom language may happen to be a specific subset of those 
languages but may enforce rules that, although would be accepted by a 
generic XML/JSON parser, are rejected by their language.  

By employing a generic parser to parse their domain-specific language, a 
developer either fails to implement the language which he needs to implement 
or is forced to develop yet another parser intended to parse some data 
structure instead of a simple stream.  While the first alternative is 
unnacceptable, the second alternative needlessly complicates things and 
imposes a considerable amount of work which otherwise the developer wouldn't 
need to deal with.



>> This means that you can
>> still employ XML or JSON in your day-to-day usage but you are obviously
>> better off developing your own parser for your specific language, due to
>> factors such as simplicity, efficiency improvements and the ability to
>> perform any sanity check right in the first parsing.
> 
> Until your language subset grows. I generally use JSON for configuration
> files and by using a generic parser, I don't have to worry about
> extending the configuration syntax.

This only represents a problem if you develop your JSON-based language so 
that it rejects unsupported language constructs, and develop your parser 
accordingly.  You don't need to do that.  Recently, I've developed a parser 
for a JSON-based document format which is designed to accept valid but 
otherwise meaningless JSON constructs.  This is easily done by adding 
generic rules which recognize generic JSON components.  With these rules in 
place, it is only a matter of setting your rules so that, when the parser 
stumbles on an unsupported but otherwise valid pattern, it simply keeps 
parsing the generic JSON document until it reaches the next component.  So, 
in the end, developing this parser ends up needing less work, being easier 
to maintain and ending up with a more efficient component than relying on 
any solution which depends on employing any generic library.


Rui Maciel

[toc] | [prev] | [next] | [standalone]

#1135

Nicolas George <nicolas$george@salle-s.org> wrote:
> Rui Maciel , dans le message
> <4e453be5$0$31423$a729d347@news.telepac.pt>, a écrit :
> > This sort of library isn't very common because it's functionality is
> > trivial to implement, and it tends to be tied to what the developer
> > intends to achieve.

> It seems trivial, but it is not at all, if you want to do things properly.
> If you write your own library, you have to document the answer to the
> following questions:

> - How do I quote strings?

> - How do I include the quotation mark inside a string?

> - Are the strings sequences of bytes or of characters? And if characters,
>   in what encoding?

> - Can I split a long string on several lines?

> - How do I insert a comment?

> - If I comment out a multi-line string, does the comment span?

> - How can I include a control character in a string?

> - etc.

Almost all of these questions are answered by using the common unix
convention of backslash- and quote-escaping, utf-8, and a comment character
which causes the parser to consume to the end-of-line. The only issue that
comes to mind is whether a quote can span across a new-line character.

All of this can be implemented using a small switch-based state machine in
roughly 30 lines of code, plus code to turn the tokens into useful data
structures; but most of the latter would need to be done with any other
system.

> Usually, with a solution designed by someone who thought it was trivial,
> the answer to a lot of these questions will be "I do not know" or "you
> can't".

> Using an existing library for a well-known syntax, such as XML or JSON,
> has the double advantage that (1) the library will usually take care of
> allowing corner cases, such as including the quotation character inside a
> string, and (2) the users will usually be already be familiar with the
> syntax.

OTOH, if your needs are simple you may end up writing more code to interface
with the library than writing everything from scratch. Not to mention the
added dependency.

OTOOH, I tend to eschew configurations files as much as possible. Most
options can be handled as switch parameters using getopt_long(). When that
becomes insufficient you're usually way past .ini land, and I prefer using
something more robust like Lua, which can be both a simple hierarchical
configuration language as well as allowing the programmer _and_ user to get
as sophisticated as necessary, including using XML or JSON.

[toc] | [prev] | [next] | [standalone]

#1136

William Ahern , dans le message
<l8dgh8-uba.ln1@wilbur.25thandClement.com>, a écrit :
> Almost all of these questions are answered by using the common unix
> convention of backslash- and quote-escaping, utf-8, and a comment character
> which causes the parser to consume to the end-of-line. The only issue that
> comes to mind is whether a quote can span across a new-line character.

Of course, the questions can be answered. That is not the problem. The
problem is: amongst developers who decide to write their own parser, how
many will first write it with stupid scanf()s and never come back to
actually implement usable escaping rules?

> All of this can be implemented using a small switch-based state machine in
> roughly 30 lines of code,

I think this is somewhat underestimated.

> OTOOH, I tend to eschew configurations files as much as possible. Most
> options can be handled as switch parameters using getopt_long(). When that
> becomes insufficient you're usually way past .ini land,

I agree. Using the environment is also a solution I like.

>							  and I prefer using
> something more robust like Lua, which can be both a simple hierarchical
> configuration language as well as allowing the programmer _and_ user to get
> as sophisticated as necessary, including using XML or JSON.

Of course, this is the next step. But there are drawbacks too. The decision
depends on the particular case.

[toc] | [prev] | [next] | [standalone]

#1137

Nicolas George <nicolas$george@salle-s.org> wrote:
> William Ahern , dans le message
<snip>
> > Almost all of these questions are answered by using the common unix
> > convention of backslash- and quote-escaping, utf-8, and a comment
> > character which causes the parser to consume to the end-of-line. The
> > only issue that comes to mind is whether a quote can span across a
> > new-line character.

> Of course, the questions can be answered. That is not the problem. The
> problem is: amongst developers who decide to write their own parser, how
> many will first write it with stupid scanf()s and never come back to
> actually implement usable escaping rules?

Conceded. But I don't want to be using such software, anyhow. If the
fundamentals of parsing escape the author, a poorly written config parser
will likely be the least of the issues in the project.

> > All of this can be implemented using a small switch-based state machine
> > in roughly 30 lines of code,

> I think this is somewhat underestimated.

I was off by roughly 25 lines (including spacing) from what I had in mind.
Admittedly, something still needs to be built on top of this, but that
depends on how the user wishes to integrate with the application. All the
escaping is handled here. NOTE: this code is entirely untested.

#define isescaped(ch) ((1<<8) & (ch))
#define isquoted(ch) ((1<<9) & (ch))

struct state {
	_Bool escaped, quoted;
}; /* struct state */

int nextchar(struct state *st, FILE *fp) {
	int ch;

	while (EOF != (ch = getc(fp))) {
		if (st.escaped)
			goto literal;

		switch (ch) {
		case '\\':
			st.escaped = 1;

			break;
		case '"':
			st.quoted = !st.quoted;

			break;
		case '#':
			if (st.quoted)
				goto literal;

			while (EOF != (ch = getc(fp)) && ch != '\n')
				;;

			return ch;
literal:	default:
			if (st.escaped) {
				ch |= 1<<8;
				st.escaped = 0;
			}

			if (st.quoted)
				ch |= 1<<9;

			return ch;
		} /* switch() */
	} /* while() */

	return EOF;
} /* nextchar() */

[toc] | [prev] | [next] | [standalone]

#1143

Nicolas George wrote:

> William Ahern , dans le message
> <l8dgh8-uba.ln1@wilbur.25thandClement.com>, a écrit :
>> Almost all of these questions are answered by using the common unix
>> convention of backslash- and quote-escaping, utf-8, and a comment
>> character which causes the parser to consume to the end-of-line. The only
>> issue that comes to mind is whether a quote can span across a new-line
>> character.
> 
> Of course, the questions can be answered. That is not the problem. The
> problem is: amongst developers who decide to write their own parser, how
> many will first write it with stupid scanf()s and never come back to
> actually implement usable escaping rules?

I can tell you exactly who will do that: those who don't know how to write a 
parser, nor are aware that they are in fact trying to write a parser.


>> All of this can be implemented using a small switch-based state machine
>> in roughly 30 lines of code,
> 
> I think this is somewhat underestimated.

I don't know if it is underestimated, but I do know that it's irrelevant.  
With the inception of tools such as flex and re2c[2], it's possible (and 
desireable) to automatically generate lexers by defining a set of regular 
expressions.  So, it doesn't really matter if a given terminal token 
requires a state machine with 30 or 3000-LoC, because you only need to 
provide a single LoC to generate the state machine for it.


>> OTOOH, I tend to eschew configurations files as much as possible. Most
>> options can be handled as switch parameters using getopt_long(). When
>> that becomes insufficient you're usually way past .ini land,
> 
> I agree. Using the environment is also a solution I like.

This isn't a either/or case.  It's quite possible and desireable to 
integrate config files, environment variables and command line arguments in 
a single component dedicated to manage a program's options.  The data is 
still the same and the only thing that varies is how you input it.  
Personally, I tend to pair all command line options with an equivalent 
config file option, and let the options passed through the command line to 
override the ones defined in the config files.  In this way, the config 
files can be used to specify long-term options, which means the user doesn't 
need to state them each time the application is executed, and is still able 
to set those options whenever he wishes to do so.


Rui Maciel

[1] http://flex.sourceforge.net/
[2] http://re2c.org/

[toc] | [prev] | [next] | [standalone]

#1147

Rui Maciel <rui.maciel@gmail.com> wrote:
<snip>
> With the inception of tools such as flex and re2c[2], it's possible (and 
> desireable) to automatically generate lexers by defining a set of regular 
> expressions.
<snip>

Ragel is much nicer than re2c, IMO. I use Ragel quite often. But sometimes
any dependency is frowned upon, especially if its only needed for such a
small and inconsequential chore as a simple config file.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.unix.programmer

csiph-web