Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Terry Reedy <tjreedy@udel.edu>
Subject: Re: The "right" way to use config files
Date: Sat, 09 Aug 2014 18:30:17 -0400
References: <ls51q0$4ea$1@speranza.aioe.org> <mailman.12797.1407605431.18130.python-list@python.org> <ls5odq$qn9$1@speranza.aioe.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0
In-Reply-To: <ls5odq$qn9$1@speranza.aioe.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.12802.1407623446.18130.python-list@python.org>
Lines: 187
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:75958

On 8/9/2014 2:14 PM, Fabien wrote:
> On 09.08.2014 19:29, Terry Reedy wrote:
>> If possible, functions should *return* their results, or yield their
>> results in chunks (as generators). Let the driver function decide where
>> to put results.  Aside from separating concerns, this makes testing much
>> easier.
>
> I see. But then this is also true for parameters, right? And yet we
> return to my original question ;-)
>
>
> Let's say my configfile looks like this:
>
> -----------------
> ### app/config.cfg
> # General params
> output_dir = '..'
> input_file = '..'
>
> # Func 1 params
> [func1]
>      enable = True
>      threshold = 0.1
>      maxite = 1
> -----------------
>
> And I have a myconfig module which looks like:
>
> -----------------
> ### app/myconfig.py
>
> import ConfigObj
>
> parser = obj() # parser will be instanciated by initialize

Try parser = object() to actually run, but the line is not needed. 
Instead put "parser: instantiated by initialize" in the docstring.
>
> def initialize(cfgfile=None):
>     global parser
>     parser = ConfigObj(cfgfile, file_error=True)
> -----------------
>
> My main program could look like this:
>
> -----------------
> ### app/mainprogram_1.py
>
> import myconfig
>
> def func1():
>      # the params are in the cfg
>      threshold = myconfig.parser['func1'].as_float('threshold')
>      maxite = myconfig.parser['func1'].as_long('maxite')
>
>      # dummy operations
>      score = 100.
>      ite = 1
>      while (score > threshold) and (ite < maxite):
>          score /= 10
>          ite += 1
>
>      # dummy return
>      return score
>
> def main():
>      myconfig.initialize(sys.argv[1])
>
>      if myconfig.parser['func1'].as_bool('enable'):
>          results = func1()
>
> if __name__ == '__main__':
>      main()
> -----------------

The advantage of TDD is that it forces one to make code testable as you 
do. Old code may not be designed to be so easily testable, as I have 
learned trying to add tests to idlelib. For the above, I would consider

def func1_algo(threshhold, maxite):  # possible separte file
     score = 100.
     ite = 1
     while (score > threshold) and (ite < maxite):
         score /= 10
         ite += 1
     return score

def func1():  # interface wrapper
     threshold = myconfig.parser['func1'].as_float('threshold')
     maxite = myconfig.parser['func1'].as_long('maxite')
     return func1_algo(threshhold, maxite)

This is a slight bit of extra work, but now you can separately test (and 
modify) the algorithm and the interfacing.  Testing the algorithm is 
easy, which encourages testing multiple i/o pairs.

for in, out in iopairs:
   assert func1_algo(in) == out  # or self.assertEqual, or ...

(or close enough for float outputs)

As for the interfacing: you can write and read multiple versions of 
config.cfg (relatively slow), use something like unittest.mock to mock 
the myconfig module, or write something fairly simple (py3 code).

class Entry(dict):
     def as_bool(self, name):
         s = self[name]
         return True if s == 'True' else False if s == 'False' else None
     def as_int(self, name):
         return int(self[name])
     as_long = as_int
     def as_float(self, name):
         return float(self[name])

class Config(object):
     def initialize(self, argv):
         pass
myconfig = Config()  # a module is like a singleton class
myconfig.initialize('a')  # test that does not raise

# In use for testing, uncomment the following two lines
# import mainprogram_1.py as mp1
# mp1.myconfig = myconfig

f1_cfg = Entry({
     'enable': 'True',
     'threshold': '0.1',
     'maxite': '1',
     })
myconfig.parser = {'func1': f1_cfg}

print(myconfig.parser['func1'].as_float('threshold') == 0.1)
print(myconfig.parser['func1'].as_long('maxite') == 1)
print(myconfig.parser['func1'].as_bool('enable') == True)

f1_cfg['maxite'] = 5
print(myconfig.parser['func1'].as_int('maxite') == 5)
# prints True 4 times

Notice that you inject the mock myconfig into the tested module just 
one. After that, you can change anything within parser or replace parser 
with a new dict.

> Or like this:
>
> -----------------
> ### app/mainprogram_2.py
>
> import myconfig
>
> def func1(threshold=None, maxite=None):

These should not have defaults; avoid extra work!

>      # dummy operations
>      score = 100.
>      ite = 1
>      while (score > threshold) and (ite < maxite):
>          score /= 10
>          ite += 1
>
>      # dummy return
>      return score
>
> def main():
>      myconfig.initialize(sys.argv[1])
>
>      if myconfig.parser['func1'].as_bool('enable'):
>          # the params are in the cfg
>          threshold = myconfig.parser['func1'].as_float('threshold')
>          maxite = myconfig.parser['func1'].as_long('maxite')
>          results = func1(threshold=threshold, maxite=maxite)
>
> if __name__ == '__main__':
>      main()
> -----------------
>
> In this case, program2 is easier to test/understand, but if the
> parameters become numerous it could be a pain...

This is equivalent to what i wrote except for putting the wrapper inline 
in main().  Testing is the same for either.

-- 
Terry Jan Reedy