Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #78478 > unrolled thread

Keepin constants, configuration values, etc. in Python - dedicated module or what?

Back to article view | Back to comp.lang.python

Contents

  Keepin constants, configuration values, etc. in Python - dedicated module or what? cl@isbd.net - 2014-09-30 12:35 +0100
    Re:Keepin constants, configuration values, etc. in Python - dedicated module or what? Dave Angel <davea@davea.name> - 2014-09-30 09:14 -0400
      Re: Re:Keepin constants, configuration values, etc. in Python - dedicated module or what? cl@isbd.net - 2014-09-30 15:55 +0100
        Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Dave Angel <d@davea.name> - 2014-10-02 14:50 -0400
    Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Rustom Mody <rustompmody@gmail.com> - 2014-09-30 06:15 -0700
      Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? cl@isbd.net - 2014-09-30 16:06 +0100
        Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? jkn <jkn_gg@nicorp.f9.co.uk> - 2014-09-30 08:59 -0700
        Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Rustom Mody <rustompmody@gmail.com> - 2014-09-30 09:39 -0700
          Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Chris Angelico <rosuav@gmail.com> - 2014-10-01 02:51 +1000
            Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Rustom Mody <rustompmody@gmail.com> - 2014-09-30 10:01 -0700
              Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Chris Angelico <rosuav@gmail.com> - 2014-10-01 03:15 +1000
                Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Rustom Mody <rustompmody@gmail.com> - 2014-09-30 10:31 -0700
                  Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Chris Angelico <rosuav@gmail.com> - 2014-10-01 03:46 +1000
                Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-02 00:49 +1000
                  Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Chris Angelico <rosuav@gmail.com> - 2014-10-02 00:55 +1000
              Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? cl@isbd.net - 2014-09-30 18:30 +0100
                Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Chris Angelico <rosuav@gmail.com> - 2014-10-01 03:49 +1000
          Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? Rustom Mody <rustompmody@gmail.com> - 2014-10-16 18:24 -0700
    Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? "Neil D. Cerutti" <neilc@norwich.edu> - 2014-09-30 12:16 -0400
      Re: Keepin constants, configuration values, etc. in Python - dedicated module or what? cl@isbd.net - 2014-09-30 18:50 +0100

#78478 — Keepin constants, configuration values, etc. in Python - dedicated module or what?

I am developing some code which runs on a (remote from me most of the
time) Beaglebone Black single board computer.  It reads various items
of data (voltages, currents, temperatures, etc.) using both a 1-wire
bus system and the Beaglebone's ADC inputs.  The values are stored
at hourly intervals into a database which is regularly rsync'ed
across to my home system where I use the values for monitoring etc.

Associated with each value are various parameters, i.e. for each ADC
input of the Beaglebone there is the ADC input number, a short name
for the value, a conversion factor and a description.  Similarly for
the 1-wire inputs there is the 1-wire filesystem filename, a short
name for the data and a longer description.

I am puzzling where and how to keep these configuration values. My
current design has them in dedicated tables in the database but this
is rather clumsy in many ways as there's an overhead reading them
every time the program needs them and changing them isn't particularly
convenient at the Beaglebone doesn't have a GUI so it has to be done
using SQL from the command line.

Does it make sense to add them to the modules which handle the reading
of the inputs?  I already have modules defining classes called Adc and
OneWire, is it a reasonable approach to add the configuration to these
as, probably, dictionaries?  Modifying it would be pretty simple -
just edit the python source (one of the easiest things for me to do on
the Beaglebone as my sole access is via ssh/command line).

Thus I'd have something like (apologies for any syntax errors):-

cfg = { "LeisureVolts": ["AIN0", 0.061256, "Leisure Battery Voltage"],
        "StarterVolts": ["AIN1", 0.060943, "Starter Battery Voltage"],
        "LeisureAmps1": ["AIN2", 0.423122, "Leisure Battery Current"}

(It might be better to makes those lists dictionaries, but it shows
the idea)

Are there any better ways of doing this?  E.g. some sort of standard
configuration file format that Python knows about?  I would actually
quite like to keep the configuration data separate from the code as it
would simplify using the data at the 'home' end of things as I'd just
need to copy the configuration file across.  This was why the database
approach appealed at first as all I need to do is copy the database
and everything is in there.

I'm not really expecting quick/glib answers, just some ideas and
comments on the various ways of doing this and their advantages and
disadvantages.

-- 
Chris Green
·

[toc] | [next] | [standalone]

#78482

cl@isbd.net Wrote in message:
> I am developing some code which runs on a (remote from me most of the
> time) Beaglebone Black single board computer.  It reads various items
> of data (voltages, currents, temperatures, etc.) using both a 1-wire
> bus system and the Beaglebone's ADC inputs.  The values are stored
> at hourly intervals into a database which is regularly rsync'ed
> across to my home system where I use the values for monitoring etc.
> 
> Associated with each value are various parameters, i.e. for each ADC
> input of the Beaglebone there is the ADC input number, a short name
> for the value, a conversion factor and a description.  Similarly for
> the 1-wire inputs there is the 1-wire filesystem filename, a short
> name for the data and a longer description.
> 
> I am puzzling where and how to keep these configuration values. My
> current design has them in dedicated tables in the database but this
> is rather clumsy in many ways as there's an overhead reading them
> every time the program needs them and changing them isn't particularly
> convenient at the Beaglebone doesn't have a GUI so it has to be done
> using SQL from the command line.

It is very useful for the database to be self contained and self
 describing.  Among other things it means that you can easily take
 a snapshot at any time,  and that snapshot is not tied to any
 specific version of the code. As for changing with sql, you can
 always add a configuration utility when using sql becomes a
 nuisance,  or introduces errors. 

The key thing is to manage change.  You need to decide which
 things are conceivably going to change and how you'll manage both
 the change itself and the inevitable split between prechange data
 and post. One kind of change might be fixing the spelling of
 battery.  No biggie,  just let new accesses get the new one. 
 Another kind might be the addition of a new instance of an adc
 converter.  Not a problem,  as long as you don't reuse an old
 name. And presumably you never remove an old name from the
 config.

More troublesome is adding a new kind of data. You have to decide
 whether it's worth generalizing the code to anticipate the
 change,  or just admit that the code will grow at that point and
 that old code won't deal with the new data. 

> 
> Does it make sense to add them to the modules which handle the reading
> of the inputs?  I already have modules defining classes called Adc and
> OneWire, is it a reasonable approach to add the configuration to these
> as, probably, dictionaries?  Modifying it would be pretty simple -
> just edit the python source (one of the easiest things for me to do on
> the Beaglebone as my sole access is via ssh/command line).
> 

Nope, read it from the db. You still might be loading it to a cfg
 variable, however. 

> Thus I'd have something like (apologies for any syntax errors):-
> 
> cfg = { "LeisureVolts": ["AIN0", 0.061256, "Leisure Battery Voltage"],
>         "StarterVolts": ["AIN1", 0.060943, "Starter Battery Voltage"],
>         "LeisureAmps1": ["AIN2", 0.423122, "Leisure Battery Current"}
> 
> (It might be better to makes those lists dictionaries, but it shows
> the idea)

Actually it's probably better to use named tuples. All those lists
 are apparently of identical length with identical meaning of a
 given element offset.  But if named tuple isn't flexible,enough, 
 you probably need a new class.

> 
> Are there any better ways of doing this?  E.g. some sort of standard
> configuration file format that Python knows about?  I would actually
> quite like to keep the configuration data separate from the code as it
> would simplify using the data at the 'home' end of things as I'd just
> need to copy the configuration file across.  This was why the database
> approach appealed at first as all I need to do is copy the database
> and everything is in there.
> 
> I'm not really expecting quick/glib answers, just some ideas and
> comments on the various ways of doing this and their advantages and
> disadvantages.

Main next question is whether you can restart each system when you
 add to the configuration.  


-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#78488

Dave Angel <davea@davea.name> wrote:
> cl@isbd.net Wrote in message:
> > I am puzzling where and how to keep these configuration values. My
> > current design has them in dedicated tables in the database but this
> > is rather clumsy in many ways as there's an overhead reading them
> > every time the program needs them and changing them isn't particularly
> > convenient at the Beaglebone doesn't have a GUI so it has to be done
> > using SQL from the command line.
> 
> It is very useful for the database to be self contained and self
>  describing.  Among other things it means that you can easily take
>  a snapshot at any time,  and that snapshot is not tied to any
>  specific version of the code. As for changing with sql, you can
>  always add a configuration utility when using sql becomes a
>  nuisance,  or introduces errors. 
> 
I guess I can write a script to do the change but it's more code to
write and maintain as opposed to simply editing a configuration file
with tools I know well.


> The key thing is to manage change.  You need to decide which
>  things are conceivably going to change and how you'll manage both
>  the change itself and the inevitable split between prechange data
>  and post. One kind of change might be fixing the spelling of
>  battery.  No biggie,  just let new accesses get the new one. 
>  Another kind might be the addition of a new instance of an adc
>  converter.  Not a problem,  as long as you don't reuse an old
>  name. And presumably you never remove an old name from the
>  config.
> 
The only things really likely to change (and may change regularly) are
the conversion factors, drifting voltage references etc. will
inevitably require these to be recalibrated every so often.  Other
than that it's just typos or me deciding to change the name of
something.


> More troublesome is adding a new kind of data. You have to decide
>  whether it's worth generalizing the code to anticipate the
>  change,  or just admit that the code will grow at that point and
>  that old code won't deal with the new data. 
> 
There's a separate module for each input type (just ADC and 1-wire at
the moment), I don't anticipate any more though I suppose there might
be digital inputs one day.  So a configuration [file] for each type
seems to make sense, generalising it would be more trouble than it's
worth I think.


> > 
> > Does it make sense to add them to the modules which handle the reading
> > of the inputs?  I already have modules defining classes called Adc and
> > OneWire, is it a reasonable approach to add the configuration to these
> > as, probably, dictionaries?  Modifying it would be pretty simple -
> > just edit the python source (one of the easiest things for me to do on
> > the Beaglebone as my sole access is via ssh/command line).
> > 
> 
> Nope, read it from the db. You still might be loading it to a cfg
>  variable, however. 
> 
> > Thus I'd have something like (apologies for any syntax errors):-
> > 
> > cfg = { "LeisureVolts": ["AIN0", 0.061256, "Leisure Battery Voltage"],
> >         "StarterVolts": ["AIN1", 0.060943, "Starter Battery Voltage"],
> >         "LeisureAmps1": ["AIN2", 0.423122, "Leisure Battery Current"}
> > 
> > (It might be better to makes those lists dictionaries, but it shows
> > the idea)
> 
> Actually it's probably better to use named tuples. All those lists
>  are apparently of identical length with identical meaning of a
>  given element offset.  But if named tuple isn't flexible,enough, 
>  you probably need a new class.
> 
> > 
> > Are there any better ways of doing this?  E.g. some sort of standard
> > configuration file format that Python knows about?  I would actually
> > quite like to keep the configuration data separate from the code as it
> > would simplify using the data at the 'home' end of things as I'd just
> > need to copy the configuration file across.  This was why the database
> > approach appealed at first as all I need to do is copy the database
> > and everything is in there.
> > 
> > I'm not really expecting quick/glib answers, just some ideas and
> > comments on the various ways of doing this and their advantages and
> > disadvantages.
> 
> Main next question is whether you can restart each system when you
>  add to the configuration.  
> 
Yes, restarting isn't a problem, the Beaglebone is dedicated to this
task and doesn't control anything so restarts are not an issue.  

-- 
Chris Green
·

[toc] | [prev] | [next] | [standalone]

#78621

On 09/30/2014 10:55 AM, cl@isbd.net wrote:
> Dave Angel <davea@davea.name> wrote:
 >>   ....
>>   name. And presumably you never remove an old name from the
>>   config.
>>
> The only things really likely to change (and may change regularly) are
> the conversion factors, drifting voltage references etc. will
> inevitably require these to be recalibrated every so often.  Other
> than that it's just typos or me deciding to change the name of
> something.
>

Right there you have an important design decision.  You then presumably 
have to have a way that each particular data point can record what 
configuration it's associated with. If you just change the calibration 
data externally, then all the old records are invalidated.

>
>> More troublesome is adding a new kind of data. You have to decide
>>   whether it's worth generalizing the code to anticipate the
>>   change,  or just admit that the code will grow at that point and
>>   that old code won't deal with the new data.
>>
> There's a separate module for each input type (just ADC and 1-wire at
> the moment), I don't anticipate any more though I suppose there might
> be digital inputs one day.  So a configuration [file] for each type
> seems to make sense, generalising it would be more trouble than it's
> worth I think.
>
>



-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#78483

On Tuesday, September 30, 2014 5:18:31 PM UTC+5:30, Chris wrote:

> I would actually
> quite like to keep the configuration data separate from the code as it
> would simplify using the data at the 'home' end of things as I'd just
> need to copy the configuration file across.  This was why the database
> approach appealed at first as all I need to do is copy the database
> and everything is in there.

Of course

> Are there any better ways of doing this?  E.g. some sort of standard
> configuration file format that Python knows about? 

Umm this is getting to be a FAQ...
Maybe it should go up somewhere?

Yes there are dozens:
- ini
- csv
- json
- yml
- xml
- pickle
- And any DBMS of your choice

I guess Ive forgotten as many as Ive listed!!

[toc] | [prev] | [next] | [standalone]

#78490

Rustom Mody <rustompmody@gmail.com> wrote:
> On Tuesday, September 30, 2014 5:18:31 PM UTC+5:30, Chris wrote:
> 
> > I would actually
> > quite like to keep the configuration data separate from the code as it
> > would simplify using the data at the 'home' end of things as I'd just
> > need to copy the configuration file across.  This was why the database
> > approach appealed at first as all I need to do is copy the database
> > and everything is in there.
> 
> Of course
> 
> > Are there any better ways of doing this?  E.g. some sort of standard
> > configuration file format that Python knows about? 
> 
> Umm this is getting to be a FAQ...
> Maybe it should go up somewhere?
> 
> Yes there are dozens:
> - ini
> - csv
> - json
> - yml
> - xml
> - pickle
> - And any DBMS of your choice
> 
> I guess Ive forgotten as many as Ive listed!!

Yes, I know, I've found most of those.  I'm really asking for help in
choosing which to use.  I think I can reject some quite quickly:-

    ini - doesn't work so well with lists/dictionaries (though possible)
    csv - rather difficult to edit
    json - one of the most likely possibilities, but prefer yml
    yml - front runner if I go for configuration files
    xml - horrible, nasty to edit, etc. I don't like XML! :-)
    pickle - not user editable as I understand it
    dbms - current solution in sqlite3 as described

What I'm really asking for is how to choose between:-

    DBMS - present solution, keeps config with data easily but more
    code and less easy to change
    yml - easy to edit config, keeps data separate from code but needs
    YAML installed and separate files to manage
    python - just keep config in the modules/classes, not easy to use
    at 'both ends' (home and remote), otherwise quite simple

My requirements are:-
    Easy to change, i.e.human readable format which can be edited
    simply as I have to do this via a terminal/ssh.
    Easy to use at local end as well as remote end, the remote end is
    where it lives and is the 'driver' as it were but I need to know
    the configuration at the local end as well.
-- 
Chris Green
·

[toc] | [prev] | [next] | [standalone]

#78491

might this be of interest (though old)?

https://wiki.python.org/moin/ConfigParserShootout

    Cheers
    Jon N

[toc] | [prev] | [next] | [standalone]

#78493

On Tuesday, September 30, 2014 8:48:15 PM UTC+5:30, c...@isbd.net wrote:
> Rustom Mody wrote:
> > On Tuesday, September 30, 2014 5:18:31 PM UTC+5:30, Chris wrote:
> > > I would actually
> > > quite like to keep the configuration data separate from the code as it
> > > would simplify using the data at the 'home' end of things as I'd just
> > > need to copy the configuration file across.  This was why the database
> > > approach appealed at first as all I need to do is copy the database
> > > and everything is in there.
> > Of course
> > > Are there any better ways of doing this?  E.g. some sort of standard
> > > configuration file format that Python knows about? 
> > Umm this is getting to be a FAQ...
> > Maybe it should go up somewhere?
> > Yes there are dozens:
> > - ini
> > - csv
> > - json
> > - yml
> > - xml
> > - pickle
> > - And any DBMS of your choice
> > I guess Ive forgotten as many as Ive listed!!

> Yes, I know, I've found most of those.  I'm really asking for help in
> choosing which to use.  I think I can reject some quite quickly:-

>     xml - horrible, nasty to edit, etc. I don't like XML! :-)

Heh! Youve proved yourself a pythonista!

>     ini - doesn't work so well with lists/dictionaries (though possible)
>     csv - rather difficult to edit

Have you tried with comma=tab?

>     yml - front runner if I go for configuration files

Yeah my favorite as well

>     json - one of the most likely possibilities, but prefer yml

Seems to be most popular nowadays -- maybe related to being almost yaml
and in the standard lib

>     pickle - not user editable as I understand it

Well not in any reasonably pleasant way!

> What I'm really asking for is how to choose between:-

<snipped>

>     python - just keep config in the modules/classes, not easy to use
>     at 'both ends' (home and remote), otherwise quite simple

Can work at a trivial level.

As soon as things get a bit larger data and code mixed up is a recipe for mess up.

[toc] | [prev] | [next] | [standalone]

#78494

On Wed, Oct 1, 2014 at 2:39 AM, Rustom Mody <rustompmody@gmail.com> wrote:
>>     python - just keep config in the modules/classes, not easy to use
>>     at 'both ends' (home and remote), otherwise quite simple
>
> Can work at a trivial level.
>
> As soon as things get a bit larger data and code mixed up is a recipe for mess up.

True, but it's certainly possible to break out the config data into an
importable module that conceptually just provides constants.
Technically it's code, yes, but it'll normally be code that looks like
your standard "name = value" config file:

# docs for first option
# more docs
# examples
# etcetera
first_option =123

# docs for second option
second_option = 234


Is that Python code, or is it a sectionless INI file, or what? There's
no difference. And you get expressions for free - simple stuff like
"7*24*60*60" to represent the number of seconds in a week (for people
who aren't intimately familiar with 604800), or calculations relative
to previous data, or whatever. Sometimes it's helpful to have just a
little code in your data.

ChrisA

[toc] | [prev] | [next] | [standalone]

#78495

On Tuesday, September 30, 2014 10:22:12 PM UTC+5:30, Chris Angelico wrote:
> On Wed, Oct 1, 2014 at 2:39 AM, Rustom Mody  wrote:
> >>     python - just keep config in the modules/classes, not easy to use
> >>     at 'both ends' (home and remote), otherwise quite simple
> > Can work at a trivial level.
> > As soon as things get a bit larger data and code mixed up is a recipe for mess up.

> True, but it's certainly possible to break out the config data into an
> importable module that conceptually just provides constants.
> Technically it's code, yes, but it'll normally be code that looks like
> your standard "name = value" config file:

> # docs for first option
> # more docs
> # examples
> # etcetera
> first_option =123

> # docs for second option
> second_option = 234

> Is that Python code, or is it a sectionless INI file, or what?

Yeah I was going to say that this is possible

> There's  no difference.

But there is! Its code that looks like data.

> And you get expressions for free - simple stuff like
> "7*24*60*60" to represent the number of seconds in a week (for people
> who aren't intimately familiar with 604800), or calculations relative
> to previous data, or whatever. Sometimes it's helpful to have just a
> little code in your data.

Not free at all. Power of code means cost of code
See http://www.w3.org/2001/tag/doc/leastPower.html

I'd reiterate though what I first said: In this case its probably
ok if the code (=data) does not cross trivial limits

[toc] | [prev] | [next] | [standalone]

#78496

On Wed, Oct 1, 2014 at 3:01 AM, Rustom Mody <rustompmody@gmail.com> wrote:
>> And you get expressions for free - simple stuff like
>> "7*24*60*60" to represent the number of seconds in a week (for people
>> who aren't intimately familiar with 604800), or calculations relative
>> to previous data, or whatever. Sometimes it's helpful to have just a
>> little code in your data.
>
> Not free at all. Power of code means cost of code
> See http://www.w3.org/2001/tag/doc/leastPower.html

It's free once you've already decided (for other reasons) to make your
config file use Python syntax, at which point you've already paid the
cost of code. But you're quite right, allowing code does have costs.
Most notably, you have to quote your strings; although the direct
benefits you get (line continuation, unambiguous handling of
leading/trailing spaces, etc) may outweigh that on their own.

> I'd reiterate though what I first said: In this case its probably
> ok if the code (=data) does not cross trivial limits

I'd agree, where "trivial limits" is defined by each individual item.
Going with straight Python code is fine for huge projects with long
config files, as long as each config entry is itself simple. You even
get a form of #include: "from otherfile import *".

ChrisA

[toc] | [prev] | [next] | [standalone]

#78497

On Tuesday, September 30, 2014 10:46:21 PM UTC+5:30, Chris Angelico wrote:
> On Wed, Oct 1, 2014 at 3:01 AM, Rustom Mody  wrote:
> >> And you get expressions for free - simple stuff like
> >> "7*24*60*60" to represent the number of seconds in a week (for people
> >> who aren't intimately familiar with 604800), or calculations relative
> >> to previous data, or whatever. Sometimes it's helpful to have just a
> >> little code in your data.
> > Not free at all. Power of code means cost of code
> > See http://www.w3.org/2001/tag/doc/leastPower.html

> It's free once you've already decided (for other reasons) to make your
> config file use Python syntax, at which point you've already paid the
> cost of code. But you're quite right, allowing code does have costs.
> Most notably, you have to quote your strings; although the direct
> benefits you get (line continuation, unambiguous handling of
> leading/trailing spaces, etc) may outweigh that on their own.

> > I'd reiterate though what I first said: In this case its probably
> > ok if the code (=data) does not cross trivial limits

> I'd agree, where "trivial limits" is defined by each individual item.
> Going with straight Python code is fine for huge projects with long
> config files, as long as each config entry is itself simple. You even
> get a form of #include: "from otherfile import *".

Well in programming we sometimes use strict/formal methods and sometimes more
informal.

In this case using configparser or pyyaml or whatever means that the config
file's syntax is rigorously fixed by that library

On the other hand if you impose a convention: Constants file-module has
NOTHING but constants, thats a non-formal convention and thats ok. 

If however you mix it up with other (real) code, you'll get a bloody mess.
This kind of stuff "7*24*60*60" is borderline and in my experience
its a slippery slope that ends up being more trouble than its worth.

Experience being emacs where because in lisp code and data are
the same, all kinds of design messes are perpetrated

[toc] | [prev] | [next] | [standalone]

#78500

On Wed, Oct 1, 2014 at 3:31 AM, Rustom Mody <rustompmody@gmail.com> wrote:
> On the other hand if you impose a convention: Constants file-module has
> NOTHING but constants, thats a non-formal convention and thats ok.
>
> If however you mix it up with other (real) code, you'll get a bloody mess.
> This kind of stuff "7*24*60*60" is borderline and in my experience
> its a slippery slope that ends up being more trouble than its worth.

Agreed, and which side of the border it's on is a matter of opinion.
Suppose you were writing an INI-style parser - would you include this
feature? I probably wouldn't include arithmetic, as it's usually more
effort than it's worth, but it isn't a bad feature inherently. I'd
definitely not include backreferencing (FOO = "bar", followed by
FOO_LEN = len(FOO) or something), or the ability to chain assignments
(FOO = BAR = 0), so those would be advised against in the convention
docs. On the other hand, a quoted string requirement ("foo\nbar")
makes a huge amount of sense in certain contexts, and I can imagine
implementing quite a lot of a programming language's string literal
syntax.

ChrisA

[toc] | [prev] | [next] | [standalone]

#78552

Chris Angelico wrote:

> I'd agree, where "trivial limits" is defined by each individual item.
> Going with straight Python code is fine for huge projects with long
> config files, as long as each config entry is itself simple. You even
> get a form of #include: "from otherfile import *".

I would argue the opposite.

If I have a large Python project, with big config files, then the added
complexity of moving the config data into separate files using a
non-executable format (say, JSON or INI) is minimal, and keeping the data
and code separate will pay off. But for small Python projects, with only a
few data values, keeping them separate is overkill.

So, to the Original Poster, I think that depending on the size of your
program and the amount of config data you have to deal with, there's
nothing wrong with including it directly in the module.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#78553

On Thu, Oct 2, 2014 at 12:49 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Chris Angelico wrote:
>
>> I'd agree, where "trivial limits" is defined by each individual item.
>> Going with straight Python code is fine for huge projects with long
>> config files, as long as each config entry is itself simple. You even
>> get a form of #include: "from otherfile import *".
>
> I would argue the opposite.

Welcome to design debates, where there are as many valid and
justifiable views as there are participants :)

Though I wasn't precluding small config files from being code - just
saying that I don't think size of project is a factor at all.

ChrisA

[toc] | [prev] | [next] | [standalone]

#78498

Rustom Mody <rustompmody@gmail.com> wrote:
> 
> > # docs for first option
> > # more docs
> > # examples
> > # etcetera
> > first_option =123
> 
> > # docs for second option
> > second_option = 234
> 
> > Is that Python code, or is it a sectionless INI file, or what?
> 
> Yeah I was going to say that this is possible
> 
> > There's  no difference.
> 
> But there is! Its code that looks like data.
> 
The main trouble with this approach is that I need some way to have
the python/config file available at the 'home' end of this as well as
at the 'remote' end.  I guess I could write a copy of the file into
the database but then I have the editing issue again, changing it
becomes messy.  If it's not in the database then how do I 'tie' it to
the data?

-- 
Chris Green
·

[toc] | [prev] | [next] | [standalone]

#78501

On Wed, Oct 1, 2014 at 3:30 AM,  <cl@isbd.net> wrote:
> The main trouble with this approach is that I need some way to have
> the python/config file available at the 'home' end of this as well as
> at the 'remote' end.  I guess I could write a copy of the file into
> the database but then I have the editing issue again, changing it
> becomes messy.  If it's not in the database then how do I 'tie' it to
> the data?

That's a design question. Maybe it's better for you to do your config
in the database. I usually find that these config files include
database credentials (server, port, user name, password), so they have
to be (a) outside the database, (b) outside source control, and (c)
separately configurable for test and production systems, even if they
run on the exact same hardware. So for those situations, it makes more
sense to have them in a script or INI file, rather than the database.
Your situation may well be completely different.

ChrisA

[toc] | [prev] | [next] | [standalone]

#79231

On Tuesday, September 30, 2014 10:09:52 PM UTC+5:30, Rustom Mody wrote:
> On Tuesday, September 30, 2014 8:48:15 PM UTC+5:30, c...@isbd.net wrote:
> > Rustom Mody wrote:
> > > On Tuesday, September 30, 2014 5:18:31 PM UTC+5:30, Chris wrote:
> > > > I would actually
> > > > quite like to keep the configuration data separate from the code as it
> > > > would simplify using the data at the 'home' end of things as I'd just
> > > > need to copy the configuration file across.  This was why the database
> > > > approach appealed at first as all I need to do is copy the database
> > > > and everything is in there.
> > > Of course
> > > > Are there any better ways of doing this?  E.g. some sort of standard
> > > > configuration file format that Python knows about? 
> > > Umm this is getting to be a FAQ...
> > > Maybe it should go up somewhere?
> > > Yes there are dozens:
> > > - ini
> > > - csv
> > > - json
> > > - yml
> > > - xml
> > > - pickle
> > > - And any DBMS of your choice
> > > I guess Ive forgotten as many as Ive listed!!

> > Yes, I know, I've found most of those.  I'm really asking for help in
> > choosing which to use.  I think I can reject some quite quickly:-

> >     xml - horrible, nasty to edit, etc. I don't like XML! :-)

> Heh! Youve proved yourself a pythonista!

> >     ini - doesn't work so well with lists/dictionaries (though possible)
> >     csv - rather difficult to edit

> Have you tried with comma=tab?

> >     yml - front runner if I go for configuration files

> Yeah my favorite as well

> >     json - one of the most likely possibilities, but prefer yml

> Seems to be most popular nowadays -- maybe related to being almost yaml
> and in the standard lib

> >     pickle - not user editable as I understand it

> Well not in any reasonably pleasant way!

> > What I'm really asking for is how to choose between:-


> >     python - just keep config in the modules/classes, not easy to use
> >     at 'both ends' (home and remote), otherwise quite simple

> Can work at a trivial level.

> As soon as things get a bit larger data and code mixed up is a recipe for mess up.

Just came across this
https://github.com/henriquebastos/python-decouple/

[toc] | [prev] | [next] | [standalone]

#78492

On 9/30/2014 7:35 AM, cl@isbd.net wrote:
> Thus I'd have something like (apologies for any syntax errors):-
>
> cfg = { "LeisureVolts": ["AIN0", 0.061256, "Leisure Battery Voltage"],
>          "StarterVolts": ["AIN1", 0.060943, "Starter Battery Voltage"],
>          "LeisureAmps1": ["AIN2", 0.423122, "Leisure Battery Current"}
>
> (It might be better to makes those lists dictionaries, but it shows
> the idea)

Using configparser.ConfigParser to read an ini-format seems like a good 
idea. I use it for my own numerous fixed format data file definitions, 
and it's been convenient and even extensible.

[LeisureVolts]
Description=Leisure Battery Voltage
Code=AIN0
Value=0.061256

[StarterVolts]
Description=Starter Battery Voltage
Code=AIN1
Value=0.060943

[LeisureAmps1]
Description=Leisure Battery Current
Code=AIN2
Value=0.423122

I've set it up so I can understand abbreviations for the field names, 
for when I want to be lazy.

Some of my values are dictionaries, which looks like this in my files 
(an alternate spelling of one of the above entries):

LeisureVolts=desc:Leisure Battery Voltage
              code:AIN2
             value:0.061256

It's simple to hook into ConfigParser to get whatever meaning you'd 
like, and whatever verification you'd find necessary.

-- 
Neil Cerutti

[toc] | [prev] | [next] | [standalone]

#78502

Neil D. Cerutti <neilc@norwich.edu> wrote:
> On 9/30/2014 7:35 AM, cl@isbd.net wrote:
> > Thus I'd have something like (apologies for any syntax errors):-
> >
> > cfg = { "LeisureVolts": ["AIN0", 0.061256, "Leisure Battery Voltage"],
> >          "StarterVolts": ["AIN1", 0.060943, "Starter Battery Voltage"],
> >          "LeisureAmps1": ["AIN2", 0.423122, "Leisure Battery Current"}
> >
> > (It might be better to makes those lists dictionaries, but it shows
> > the idea)
> 
> Using configparser.ConfigParser to read an ini-format seems like a good 
> idea. I use it for my own numerous fixed format data file definitions, 
> and it's been convenient and even extensible.
> 
> [LeisureVolts]
> Description=Leisure Battery Voltage
> Code=AIN0
> Value=0.061256
> 
> [StarterVolts]
> Description=Starter Battery Voltage
> Code=AIN1
> Value=0.060943
> 
> [LeisureAmps1]
> Description=Leisure Battery Current
> Code=AIN2
> Value=0.423122
> 
That's OK except that it doesn't associate the different sections in
any way, I need another level which includes these to indicate that
these are values read from the ADC.  There's another collection of
config data for values read from the 1-wire bus.

I guess I could have two config files, one for ADC and one for 1-wire.

In fact I think that's the best approach so far.  Ini format files are
human readable (and more to the point editable), I can store them in
the same directory as the database so they will get carried around
with the data and the configparser module is built-in.

-- 
Chris Green
·

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web