Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #94894 > unrolled thread

Most Pythonic way to store (small) configuration

Back to article view | Back to comp.lang.python

Contents

  Most Pythonic way to store (small) configuration Cecil Westerhof <Cecil@decebal.nl> - 2015-08-02 12:11 +0200
    Re: Most Pythonic way to store (small) configuration Chris Angelico <rosuav@gmail.com> - 2015-08-02 20:49 +1000
    Re: Most Pythonic way to store (small) configuration Ben Finney <ben+python@benfinney.id.au> - 2015-08-02 21:54 +1000
      Re: Most Pythonic way to store (small) configuration Cecil Westerhof <Cecil@decebal.nl> - 2015-08-02 18:51 +0200
        Re: Most Pythonic way to store (small) configuration Lele Gaifax <lele@metapensiero.it> - 2015-08-02 22:02 +0200
        Re: Most Pythonic way to store (small) configuration Cameron Simpson <cs@zip.com.au> - 2015-08-03 08:49 +1000
        Re: Most Pythonic way to store (small) configuration Ben Finney <ben+python@benfinney.id.au> - 2015-08-03 11:16 +1000
    Re: Most Pythonic way to store (small) configuration Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-08-02 16:12 +0100
    Re: Most Pythonic way to store (small) configuration Tim Chase <python.list@tim.thechases.com> - 2015-08-02 16:11 -0500
      Re: Most Pythonic way to store (small) configuration Dan Sommers <dan@tombstonezero.net> - 2015-08-03 04:02 +0000
        Re: Most Pythonic way to store (small) configuration Steven D'Aprano <steve@pearwood.info> - 2015-08-03 23:38 +1000
          Re: Most Pythonic way to store (small) configuration Chris Angelico <rosuav@gmail.com> - 2015-08-03 23:46 +1000
          Re: Most Pythonic way to store (small) configuration Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-08-03 15:37 +0100
    Re: Most Pythonic way to store (small) configuration marco.nawijn@colosso.nl - 2015-08-04 07:53 -0700
      Re: Most Pythonic way to store (small) configuration Irmen de Jong <irmen.NOSPAM@xs4all.nl> - 2015-08-04 19:06 +0200
        Re: Most Pythonic way to store (small) configuration marco.nawijn@colosso.nl - 2015-08-04 11:37 -0700
      Re: Most Pythonic way to store (small) configuration Ben Finney <ben+python@benfinney.id.au> - 2015-08-05 05:59 +1000
        Re: Most Pythonic way to store (small) configuration Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-08-05 18:32 +1000
          Re: Most Pythonic way to store (small) configuration Chris Angelico <rosuav@gmail.com> - 2015-08-05 20:01 +1000
      Re: Most Pythonic way to store (small) configuration Michael Torrie <torriem@gmail.com> - 2015-08-04 19:32 -0600
        Re: Most Pythonic way to store (small) configuration Grant Edwards <invalid@invalid.invalid> - 2015-08-05 14:00 +0000
      Re: Most Pythonic way to store (small) configuration random832@fastmail.us - 2015-08-04 22:44 -0400
      Re: Most Pythonic way to store (small) configuration Michael Torrie <torriem@gmail.com> - 2015-08-04 22:48 -0600
        Re: Most Pythonic way to store (small) configuration Rustom Mody <rustompmody@gmail.com> - 2015-08-04 21:55 -0700
          Re: Most Pythonic way to store (small) configuration Lele Gaifax <lele@metapensiero.it> - 2015-08-05 08:54 +0200
    Re: Most Pythonic way to store (small) configuration Tim Chase <python.list@tim.thechases.com> - 2015-08-05 08:18 -0500
      Re: Most Pythonic way to store (small) configuration Rustom Mody <rustompmody@gmail.com> - 2015-08-05 06:37 -0700
        Re: Most Pythonic way to store (small) configuration Tim Chase <python.list@tim.thechases.com> - 2015-08-05 15:55 -0500
          Re: Most Pythonic way to store (small) configuration Marko Rauhamaa <marko@pacujo.net> - 2015-08-06 00:47 +0300
            Re: Most Pythonic way to store (small) configuration Tim Chase <python.list@tim.thechases.com> - 2015-08-05 18:43 -0500
            Re: Most Pythonic way to store (small) configuration Chris Angelico <rosuav@gmail.com> - 2015-08-06 10:07 +1000
              Re: Most Pythonic way to store (small) configuration Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-08-06 17:33 +1000
                Re: Most Pythonic way to store (small) configuration Chris Angelico <rosuav@gmail.com> - 2015-08-06 17:51 +1000
          Re: Most Pythonic way to store (small) configuration Rustom Mody <rustompmody@gmail.com> - 2015-08-05 18:01 -0700
            Re: Most Pythonic way to store (small) configuration Rustom Mody <rustompmody@gmail.com> - 2015-08-05 18:06 -0700
    Re: Most Pythonic way to store (small) configuration Rustom Mody <rustompmody@gmail.com> - 2015-08-05 06:46 -0700
      Re: Most Pythonic way to store (small) configuration Steven D'Aprano <steve@pearwood.info> - 2015-08-06 00:08 +1000
        Re: Most Pythonic way to store (small) configuration Rustom Mody <rustompmody@gmail.com> - 2015-08-05 07:25 -0700

Page 1 of 2  [1] 2  Next page →

#94894 — Most Pythonic way to store (small) configuration

There are a lot of ways to store configuration information:
- conf file
- xml file
- database
- json file
- and possible a lot of other ways

I want to write a Python program to display cleaned log files. I do
not think I need a lot of configuration to be stored:
- some things relating to the GUI
- default behaviour
- default directory
- log files to display, including some info
  - At least until where it was displayed

Because of this I think a human readable file would be best.
Personally I do not find XML very readable. So a conf or json file
looks the most promising to me. And I would have a slight preference
for a json file.

Any comments, thoughts or tips?

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [next] | [standalone]

#94895

On Sun, Aug 2, 2015 at 8:11 PM, Cecil Westerhof <Cecil@decebal.nl> wrote:
> Because of this I think a human readable file would be best.
> Personally I do not find XML very readable. So a conf or json file
> looks the most promising to me. And I would have a slight preference
> for a json file.
>
> Any comments, thoughts or tips?

I'd agree with your analysis. XML is not readable; the two best
options would be JSON, if you need the potential for deep structure,
or the simple config file ("INI file") if you don't.

ChrisA

[toc] | [prev] | [next] | [standalone]

#94897

Cecil Westerhof <Cecil@decebal.nl> writes:

> Because of this I think a human readable file would be best.

I agree with that criterion; in the absence of compelling reasons
otherwise, human-readable and -editable text is a good default.

> Personally I do not find XML very readable. So a conf or json file
> looks the most promising to me. And I would have a slight preference
> for a json file.

XML and JSON should both be considered data serialisation formats only.

JSON is human-readable to an extent, but it is quite brittle, and there
are no comments permitted in the syntax.

So, both XML and JSON should be considered write-only, and produced only
for consumption by a computer; they are a poor choice for presenting to
a human.

The “INI” format as handled by the Python ‘configparser’ module is what
I would recommend for a simple flat configuration file. It is more
intuitive to edit, and has a conventional commenting format.

If it were more complex I might recommend YAML, which is a very
widespread, flexible hierarchical format, that still allows fairly
natural editing and commenting. Its main downside (as opposed to INI
format) is that the Python standard library does not support it.

-- 
 \         “Simplicity and elegance are unpopular because they require |
  `\           hard work and discipline to achieve and education to be |
_o__)                                appreciated.” —Edsger W. Dijkstra |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#94899

On Sunday  2 Aug 2015 13:54 CEST, Ben Finney wrote:

> Cecil Westerhof <Cecil@decebal.nl> writes:
>
>> Because of this I think a human readable file would be best.
>
> I agree with that criterion; in the absence of compelling reasons
> otherwise, human-readable and -editable text is a good default.
>
>> Personally I do not find XML very readable. So a conf or json file
>> looks the most promising to me. And I would have a slight
>> preference for a json file.
>
> XML and JSON should both be considered data serialisation formats
> only.
>
> JSON is human-readable to an extent, but it is quite brittle, and
> there are no comments permitted in the syntax.
>
> So, both XML and JSON should be considered write-only, and produced
> only for consumption by a computer; they are a poor choice for
> presenting to a human.
>
> The “INI” format as handled by the Python ‘configparser’ module is
> what I would recommend for a simple flat configuration file. It is
> more intuitive to edit, and has a conventional commenting format.

Well, I would use nested data. (A file will have extra fields besides
the name.) That is why I was thinking about json. But I will look into
it.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#94902

Cecil Westerhof <Cecil@decebal.nl> writes:

> Well, I would use nested data. (A file will have extra fields besides
> the name.) That is why I was thinking about json. But I will look into
> it.

An alternative, very similar to JSON but with some good cherries picked from
YAML is AXON, which is my preferite these days for this kind of things.
http://intellimath.bitbucket.org/axon/

ciao, lele.
-- 
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it  |                 -- Fortunato Depero, 1929.

[toc] | [prev] | [next] | [standalone]

#94904

On 02Aug2015 18:51, Cecil Westerhof <Cecil@decebal.nl> wrote:
>On Sunday  2 Aug 2015 13:54 CEST, Ben Finney wrote:
>> Cecil Westerhof <Cecil@decebal.nl> writes:
>>> Because of this I think a human readable file would be best.
>>
>> I agree with that criterion; in the absence of compelling reasons
>> otherwise, human-readable and -editable text is a good default.
[...]
>> The “INI” format as handled by the Python ‘configparser’ module is
>> what I would recommend for a simple flat configuration file. It is
>> more intuitive to edit, and has a conventional commenting format.
>
>Well, I would use nested data. (A file will have extra fields besides
>the name.) That is why I was thinking about json. But I will look into
>it.

Like others, I also recommend an INI file. You can always move to something 
more complex (and harder to edit) if it doesn't work out.

Note that "nested data" does not rule out an INI file unless you mean 
"recursive" data or quite a deep nesting of structure. Shallow structure (one 
or two levels) is very easy to embed in the INI format.

Eg:

  [main]
  this = that
  size = 10
  directory = /path/to/default/dir

  [gui]
  foreground = green
  style = baroque

  [*.log]
  stuff about log files...

  [/path/to/file1]
  title = something
  param1 = 5
  param2 = 6
  fields = column2, column7

  [column2:/path/to/file1]
  format = numeric
  scale = logarithmic

  [column7:/path/to/file1]
  format = roman
  scale = linear

See that by being a little tricky about the section names you can incorporate a 
fair degree of structure? I would not advocating going too fair down that 
rabbit hole, but for basic stuff it works well.

And of course you can put structure within a section:

  [foo]
  complexity = {'blah': 'blah blah', 'frib': {'frob': 1, 'frab': 2}}
  format_width = 10
  format_style = right-justified
  format_dialect = quoted

i.e. you could put (small) JSON snippet in some fields, or present a few 
parameters to describe "format".

There are several ways to do this kind of thing before reaching for more 
complex syntaxes. Just don't let it get too weird - that would probably be a 
sign you do want to reach for JSON or YAML.

But start with INI. Simple and easy.

Cheers,
Cameron Simpson <cs@zip.com.au>

A squealing tire is a happy tire.
        - Bruce MacInnes, Skip Barber Driving School instructor

[toc] | [prev] | [next] | [standalone]

#94910

Cecil Westerhof <Cecil@decebal.nl> writes:

> On Sunday  2 Aug 2015 13:54 CEST, Ben Finney wrote:
>
> > So, both XML and JSON should be considered write-only, and produced
> > only for consumption by a computer; they are a poor choice for
> > presenting to a human.
>
> Well, I would use nested data. (A file will have extra fields besides
> the name.)

In that case, your needs are more complex than “store some simple
configuration”. You need a human-editable format (so, not JSON and not
XML) which handles structured data well, and is widely implemented.

For nested configuration data, you would be better served by YAML
<URL:https://en.wikipedia.org/wiki/YAML>, and the PyYAML library
<URL:https://pypi.python.org/pypi/PyYAML/> is the one to use.

-- 
 \                                        學而不思則罔，思而不學則殆。 |
  `\        (To study and not think is a waste. To think and not study |
_o__)             is dangerous.) —孔夫子 Confucius (551 BCE – 479 BCE) |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#94898

On 02/08/2015 12:54, Ben Finney wrote:
> Cecil Westerhof <Cecil@decebal.nl> writes:
>
>> Because of this I think a human readable file would be best.
>
> The “INI” format as handled by the Python ‘configparser’ module is what
> I would recommend for a simple flat configuration file. It is more
> intuitive to edit, and has a conventional commenting format.

+1

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#94912

On 2015-08-02 21:54, Ben Finney wrote:
> So, both XML and JSON should be considered write-only, and produced
> only for consumption by a computer; they are a poor choice for
> presenting to a human.
> 
> The “INI” format as handled by the Python ‘configparser’ module is
> what I would recommend for a simple flat configuration file. It is
> more intuitive to edit, and has a conventional commenting format.

I second Ben's thoughts against XML & JSON -- they *can* be edited by
hand, but put the onus on the user to make perfect XML/JSON.  Config
files (".ini") are more forgiving.

However, the .ini format (or at least the stdlib implementation in
ConfigParser.py) is not without its faults, mostly when you read a
file, then write it back out:

 - comments and blank lines get lost in the process:

    [section]

    # set to local configuration
    location=path/to/foo

  will get written out as

    [section]
    location=path/to/foo
 

 - the order of options is not preserved:

    [section]
    thing=1
    other=2

   may get written back out as

    [section]
    other=2
    thing=1

   though this has improved once ConfigParser started attempting to
   use an OrderedDict by default for internal storage.

 - a single key can only appear once in a section:

    [section]
    option=one
    option=two

  gets written back out as 

    [section]
    option=two

  - implicit encoding (is it UTF-8, Latin-1, etc?)

When you understand that the underlying internal storage is a dict
(ordered or unordered, depending on availability), a lot of the above
makes sense.  But it still makes me wish for the power of git's
config-file format that seems to preserve original config files much
better.

An additional option is using a sqlite database.  The sqlite
library is part of the stdlib, and advantages include being a single
file, expandability, consistent/reliable character encoding,
cross-platform portability, and atomicity (utilities that read/write
are blocked from getting/creating incomplete data seen by the other
file).

-tkc

[toc] | [prev] | [next] | [standalone]

#94916

On Sun, 02 Aug 2015 16:11:14 -0500, Tim Chase wrote:

> On 2015-08-02 21:54, Ben Finney wrote:

>> So, both XML and JSON should be considered write-only, and produced
>> only for consumption by a computer; they are a poor choice for
>> presenting to a human.

[snip]

> I second Ben's thoughts against XML & JSON -- they *can* be edited by
> hand, but put the onus on the user to make perfect XML/JSON.  Config
> files (".ini") are more forgiving.

[snip]

> An additional option is using a sqlite database.  The sqlite library
> is part of the stdlib, and advantages include being a single file,
> expandability, consistent/reliable character encoding, cross-platform
> portability, and atomicity (utilities that read/write are blocked from
> getting/creating incomplete data seen by the other file).

Well, I have at least some non-zero chance of reading and writing JSON
or XML by hand.  Can the same be said for a sqlite database?  ;-)

[toc] | [prev] | [next] | [standalone]

#94936

On Mon, 3 Aug 2015 02:02 pm, Dan Sommers wrote:

> Well, I have at least some non-zero chance of reading and writing JSON
> or XML by hand.  Can the same be said for a sqlite database?


Real programmers edit their SQL databases directly on the hard drive platter
using a magnetised needle and a steady hand.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#94937

On Mon, Aug 3, 2015 at 11:38 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> On Mon, 3 Aug 2015 02:02 pm, Dan Sommers wrote:
>
>> Well, I have at least some non-zero chance of reading and writing JSON
>> or XML by hand.  Can the same be said for a sqlite database?
>
>
> Real programmers edit their SQL databases directly on the hard drive platter
> using a magnetised needle and a steady hand.

REAL programmers use emacs, with the C-x M-x M-butterfly command.

ChrisA

[toc] | [prev] | [next] | [standalone]

#94942

On 03/08/2015 14:38, Steven D'Aprano wrote:
> On Mon, 3 Aug 2015 02:02 pm, Dan Sommers wrote:
>
>> Well, I have at least some non-zero chance of reading and writing JSON
>> or XML by hand.  Can the same be said for a sqlite database?
>
> Real programmers edit their SQL databases directly on the hard drive platter
> using a magnetised needle and a steady hand.
>

I'd stick with SQLiteStudio or something similar if I were you.  The 
last time I tried the above, having consumed 10 bottles of Newcastle 
Brown, the results were disastrous.  I pricked my finger.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#94967

On Sunday, August 2, 2015 at 12:14:51 PM UTC+2, Cecil Westerhof wrote:
> There are a lot of ways to store configuration information:
> - conf file
> - xml file
> - database
> - json file
> - and possible a lot of other ways
> 
> I want to write a Python program to display cleaned log files. I do
> not think I need a lot of configuration to be stored:
> - some things relating to the GUI
> - default behaviour
> - default directory
> - log files to display, including some info
>   - At least until where it was displayed
> 
> Because of this I think a human readable file would be best.
> Personally I do not find XML very readable. So a conf or json file
> looks the most promising to me. And I would have a slight preference
> for a json file.
> 
> Any comments, thoughts or tips?
> 
> -- 
> Cecil Westerhof
> Senior Software Engineer
> LinkedIn: http://www.linkedin.com/in/cecilwesterhof

Why not use Python files itself as configuration files?
I have done so several times and I am very pleased with
it. I have tried both INI and JSON, but in the end always
settled on using Python itself. If I have to share the
configuration information with other languages (like C++
or Javascript) I use JSON as the 

For those who are interested, I provide a short summary
below on how I handle configuration files.

The scenario I want to support is that there is a hierarchical
set of configuration files (as typically found in Unix like
systems) that can be stored in a system wide folder (like 
/etc/app), in a user folder (like /home/user/.config/app) and
in a local folder.

Each configuration file has a fixed name, not necessarily ending
in ".py". So for example, the name of the configuration file
is "config.myapp".

Next I start looking for the existence of these files and once
I find them I start merging all the configuration settings into
one Python dictionary. The function that I use can be found below.  
(you need proper import statements, but this is standard Python).

def load_config(config_file, checked=False):
    '''Small utility function to load external configuration files.
    
    '''
    if checked is False:
        if not isfile(config_file) or islink(config_file):
            msg = 'Config file "%s" does not exist.'
            LOGGER.warning(msg)
            return {}

    # Load the module object; we use the imp.load_source function because
    # the module configuration typically has a non-standard file extension
    # (by default the file is called "config.myapp")

    module_name = 'm%s' % str(uuid.uuid1()).replace('-', '')

    # Prevent bytecode generation for configuration files
    sys.dont_write_bytecode = True

    config_module = imp.load_source(module_name, config_file)

    # Continue with bytecode generation for other normal modules
    sys.dont_write_bytecode = False

    if not hasattr(config_module, '__config__'):
        msg = 'Missing "__config__" attribute in configuration file.'
        LOGGER.warning(msg)
        return {}

    else:
        return config_module.__config__

[toc] | [prev] | [next] | [standalone]

#94970

On 4-8-2015 16:53, marco.nawijn@colosso.nl wrote:
> Why not use Python files itself as configuration files?

It could create a security risk if the config files are user-editable.
(it will make it easy to inject code into your application)

Irmen

[toc] | [prev] | [next] | [standalone]

#94972

On Tuesday, August 4, 2015 at 7:06:33 PM UTC+2, Irmen de Jong wrote:
> On 4-8-2015 16:53, marco.nawijn@colosso.nl wrote:
> > Why not use Python files itself as configuration files?
> 
> It could create a security risk if the config files are user-editable.
> (it will make it easy to inject code into your application)
> 
> Irmen

Yes, I am aware of the security risk, but right now I am not too
worried about it. To be honest, JSON would be my preferred 
configuration file format, but as others already mentioned, there is
no standard way of adding comments to a JSON file. 

Marco

[toc] | [prev] | [next] | [standalone]

#94973

marco.nawijn@colosso.nl writes:

> Why not use Python files itself as configuration files?

Because configuration data will be user-editable. (If it's not
user-editable, that is itself a poor design choice.)

If you allow executable code to be user-edited, that opens your program
to arbitrary injection of executable code. Your program becomes wide
open for security exploits, whether through malicious or accidental
bugs, and simple human error can lead to arbitrary-scope damage to the
user's system.

On another dimension, configuration files specifying the behaviour of
the system are much more useful if their format is easily parsed and
re-worked by tools the user chooses.

Your program should not be the only tool (and Python should not be the
only language) that can read and/or write the configuration data with
straightfoward data manipulation.

So a complex full-blown programming language like Python is a poor
choice for configuration data for that reason, too.

Much better to choose a tightly-defined, limited-scope configuration
data format that you can be confident any user cannot employ to run
arbitrary code.

-- 
 \     “For myself, I am an optimist — it does not seem to be much use |
  `\              being anything else.” —Winston Churchill, 1954-11-09 |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#95004

On Wednesday 05 August 2015 05:59, Ben Finney wrote:

> marco.nawijn@colosso.nl writes:
> 
>> Why not use Python files itself as configuration files?
> 
> Because configuration data will be user-editable. (If it's not
> user-editable, that is itself a poor design choice.)
> 
> If you allow executable code to be user-edited, that opens your program
> to arbitrary injection of executable code. Your program becomes wide
> open for security exploits, whether through malicious or accidental
> bugs, and simple human error can lead to arbitrary-scope damage to the
> user's system.

Yeah... it's almost as if you allowed the user to edit the source code of 
your application, or even write their own code. And nobody would do that! 

*wink*

I'm not entirely disagreeing, just adding some nuance to the discussion.

My own personal feeling is that using code as config is a little 
disquieting. It's a bit of a code smell. Do you really need that much power 
just to allow people to set some configuration settings? Using a Turing 
Complete programming language just to set a bunch of name:value pairs seems 
to be gross overkill.

But on the other hand, config systems do tend to grow in power. People often 
want conditional and computed settings. Rather than invent your own (buggy) 
mini-language to allow conf like this:

if $PLATFORM = 'Windows' set location = 'C:\The Stuff';
if $PLATFORM = 'Linux' set location = $baselocation + '/thestuff';

using Python seems like a much better idea.

Code smell or not, I don't think that Python code as config is that much of 
a problem, for most applications. Yes, the user can insert arbitrary code 
into their config file, and have it run... but it's their user account 
running it, they presumably could just insert that code into a file and run 
it with python regardless. There's (usually) no security implications.

Although I can think of some exceptions. For example, the company I work for 
has a Linux desktop designed for untrusted, hostile users (prisoners in 
jail), and using .py files as config would *definitely* not be allowed for 
that. But I think that's an exceptional case.


> On another dimension, configuration files specifying the behaviour of
> the system are much more useful if their format is easily parsed and
> re-worked by tools the user chooses.
> 
> Your program should not be the only tool (and Python should not be the
> only language) that can read and/or write the configuration data with
> straightfoward data manipulation.

Python is an open standard that anyone can use, or write their own should 
they wish. If you don't like CPython, use Jython or PyPy or 
MyFirstPythonInterpreter to pass the config data. 

A better argument may be that *running arbitrary code* should not be 
required in order to parse a config file. I'm sympathetic to that argument 
too. But if that's your worry, and you absolutely must read a .py config 
file, you could write your own stripped down interpreter of a limited subset 
of the language, and use that. No, it won't be able to determine all the 
config settings from arbitrary .py files, but it may give you a practical 
solution which is good enough for what you need.


> So a complex full-blown programming language like Python is a poor
> choice for configuration data for that reason, too.
> 
> Much better to choose a tightly-defined, limited-scope configuration
> data format that you can be confident any user cannot employ to run
> arbitrary code.

Sure, if your config requirements are met by such a data format.


-- 
Steve

[toc] | [prev] | [next] | [standalone]

#95009

On Wed, Aug 5, 2015 at 6:32 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> My own personal feeling is that using code as config is a little
> disquieting. It's a bit of a code smell. Do you really need that much power
> just to allow people to set some configuration settings? Using a Turing
> Complete programming language just to set a bunch of name:value pairs seems
> to be gross overkill.
>
> But on the other hand, config systems do tend to grow in power. People often
> want conditional and computed settings. Rather than invent your own (buggy)
> mini-language to allow conf like this:
>
> if $PLATFORM = 'Windows' set location = 'C:\The Stuff';
> if $PLATFORM = 'Linux' set location = $baselocation + '/thestuff';
>
> using Python seems like a much better idea.

I often have config files where the example is really simple, like this:

https://github.com/Rosuav/LetMeKnow/blob/master/keys_sample.py
https://github.com/Rosuav/Yosemite/blob/master/config.py

but the in-production versions sometimes have a teensy bit more code
in them - maybe an if-expression, maybe pulling something from
os.environ. In both of those cases, any person editing the config file
will also have full power to tamper with the source code, so the
security issue doesn't apply; and apart from requiring quotes around
string values, it's pretty much the level of simplicity you'd get out
of any other config file format. You get comments, you get freedom to
add whitespace anywhere you like (or not add it, if you so desire),
it's great! Right? Well, there is another subtlety, and that's that
there's no error checking. If you duplicate an entry, nobody will
issue even a warning. For small config files, that's fine; but what if
you're editing something the size of postgresql.conf? You can't keep
it all on screen or in your head, and if you simply search the file
for some keyword, would you notice that there are two of them?

So, I'd recommend this if - and ONLY if - the files are normally going
to be small enough to fit on a single page. Otherwise, consider
carefully whether you'd later on want to add error checking like that,
and consider how hard it would be to hack that in.

Drink responsibly.

ChrisA

[toc] | [prev] | [next] | [standalone]

#94983

On 08/04/2015 01:59 PM, Ben Finney wrote:
> marco.nawijn@colosso.nl writes:
> 
>> Why not use Python files itself as configuration files?
> 
> Because configuration data will be user-editable. (If it's not
> user-editable, that is itself a poor design choice.)
> 
> If you allow executable code to be user-edited, that opens your program
> to arbitrary injection of executable code. Your program becomes wide
> open for security exploits, whether through malicious or accidental
> bugs, and simple human error can lead to arbitrary-scope damage to the
> user's system.

We need to state the context here.  The only context in which having a
Python config file is dangerous is when the python program runs as a
different user/privilege than the owner of the config file.  If the user
owns the python files as well as the config file then none of this matters.

In most cases, I've never bought the argument you and others are making
here about security and demanding yet another DSL.  In fact I find the
argument to be rather circular in that we're dealing with programs that
aren't compiled but written in Python anyway. I can open and edit any
python file in the project that I want and make arbitrary, possibly
"malicious" changes to it! Oh no!  The idea that a malicious user could
inject python code in this instance and somehow deliberately harm the
system is kind of silly if you think about it.  It's me that's running
the python code in the first place. I could open any file and change it.
 I'm already running arbitrary code.

If I'm talking about a system service that is doing things for non-root
users, then yeah I'll agree with your argument completely.  But not for
most other situations.  Even a system service, if the config file is
owned by root, I'm okay with using python as configuration.  Because if
root's compromised, all bets are off anyway and all the python scripts
could be modified.

In fact python as configuration works very well for my purposes, and it
works well for Django, and there are many other projects that also do
this.  I don't think you'd want to use any other mechanism for
configuring Django, frankly.  I'm a bit surprised no one has mentioned
Django in this discussion yet.

In many of my projects I put basic config variables in a file like
config.py and import that in each module that needs it.  The config
module doubles as a global namespace for sharing between modules as well.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web