Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #32593 > unrolled thread

Organisation of python classes and their methods

Back to article view | Back to comp.lang.python

Contents

  Organisation of python classes and their methods Martin Hewitson <martinhewitson@mac.com> - 2012-11-02 07:16 +0100
    Re: Organisation of python classes and their methods Paul Rubin <no.email@nospam.invalid> - 2012-11-02 00:38 -0700
      Re: Organisation of python classes and their methods Martin Hewitson <martinhewitson@mac.com> - 2012-11-02 09:08 +0100
        Re: Organisation of python classes and their methods Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-11-02 11:18 +0100
          Re: Organisation of python classes and their methods Martin Hewitson <martinhewitson@me.com> - 2012-11-02 15:49 +0100
          Re: Organisation of python classes and their methods Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-11-02 17:02 +0000
        Re: Organisation of python classes and their methods Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-03 01:38 +0000
      Re: Organisation of python classes and their methods Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-11-02 08:40 +0000
        Re: Organisation of python classes and their methods Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-02 10:20 +0000
      Re: Organisation of python classes and their methods Chris Angelico <rosuav@gmail.com> - 2012-11-02 20:15 +1100
      Re: Organisation of python classes and their methods Martin Hewitson <martinhewitson@mac.com> - 2012-11-02 09:45 +0100
      Re: Organisation of python classes and their methods Peter Otten <__peter__@web.de> - 2012-11-02 11:21 +0100
      Re: Organisation of python classes and their methods Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-11-02 10:48 +0000
      Re: Organisation of python classes and their methods Robert Kern <robert.kern@gmail.com> - 2012-11-02 10:55 +0000
      Re: Organisation of python classes and their methods Robert Kern <robert.kern@gmail.com> - 2012-11-02 11:07 +0000
    Re: Organisation of python classes and their methods Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-03 01:06 +0000
      Re: Organisation of python classes and their methods Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-03 01:08 +0000

#32593 — Organisation of python classes and their methods

Dear list,

I'm relatively new to Python and have googled and googled but haven't found a reasonable answer to this question, so I thought I'd ask it here.

I'm beginning a large Python project which contains many packages, modules and classes. The organisation of those is clear to me.

Now, the classes can contain many methods (100s of data analysis methods) which operate on instances of the class they belong to. These methods can be long and complex. So if I put these methods all in the module file inside the class, the file will get insanely long. Reading on google, the answer is usually "refactor", but that really doesn't make sense here. It's just that the methods are many, and each method can be a long piece of code. So, is there a way to put these methods in their own files and have them 'included' in the class somehow? I read a little about mixins but all the solutions looked very hacky. Is there an official python way to do this? I don't like having source files with 100's of lines of code in, let alone 1000's.

Many thanks,

Martin

[toc] | [next] | [standalone]

#32594

Martin Hewitson <martinhewitson@mac.com> writes:
> So, is there a way to put these methods in their own files and have
> them 'included' in the class somehow? ... Is there an official python
> way to do this? I don't like having source files with 100's of lines
> of code in, let alone 1000's.

That code sounds kind of smelly... why are there so many methods per
class?  

Python lets you inject new methods into existing classes (this is
sometimes called duck punching) but I don't recommend doing this.

A few hundred lines of code in a file is completely reasonable.  Even a
few thousand is ok.  IME it starts getting unwieldy at 3000 or so.

[toc] | [prev] | [next] | [standalone]

#32597

On 2, Nov, 2012, at 08:38 AM, Paul Rubin <no.email@nospam.invalid> wrote:

> Martin Hewitson <martinhewitson@mac.com> writes:
>> So, is there a way to put these methods in their own files and have
>> them 'included' in the class somehow? ... Is there an official python
>> way to do this? I don't like having source files with 100's of lines
>> of code in, let alone 1000's.
> 
> That code sounds kind of smelly... why are there so many methods per
> class?  

Simply because there are many different ways to process the data. The class encapsulates the data, and the user can process the data in many ways. Of course, one could have classes which encapsulate the algorithms, as well as the data, but it also seems natural to me to have algorithms as methods which are part of the data class, so the user operates on the data using methods of that class. 

> 
> Python lets you inject new methods into existing classes (this is
> sometimes called duck punching) but I don't recommend doing this.

Is there not a way just to declare the method in the class and put the actual implementation in another file on the python path so that it's picked up a run time?

> 
> A few hundred lines of code in a file is completely reasonable.  Even a
> few thousand is ok.  IME it starts getting unwieldy at 3000 or so.

In all other projects (in other languages) I've always tried to keep individual files short to ease maintainability and search-ability (if there is such a word).

Even if one takes reasonable numbers: 20 methods, each method has 20 lines of documentation, then we immediately have 400 lines in the file before writing a line of code. It would seem much more natural to me to have these methods in their own file, grouped nicely in sub-directories. But it seems this is not the python way. Sigh.


Thanks for your thoughts,

Martin

[toc] | [prev] | [next] | [standalone]

#32622

Am 02.11.2012 09:08, schrieb Martin Hewitson:
> On 2, Nov, 2012, at 08:38 AM, Paul Rubin <no.email@nospam.invalid>
> wrote:
>> Martin Hewitson <martinhewitson@mac.com> writes:
>>> So, is there a way to put these methods in their own files and
>>> have them 'included' in the class somehow? ... Is there an
>>> official python way to do this? I don't like having source files
>>> with 100's of lines of code in, let alone 1000's.
>>
>> That code sounds kind of smelly... why are there so many methods
>> per class?
>
> Simply because there are many different ways to process the data. The
> class encapsulates the data, and the user can process the data in
> many ways. Of course, one could have classes which encapsulate the
> algorithms, as well as the data, but it also seems natural to me to
> have algorithms as methods which are part of the data class, so the
> user operates on the data using methods of that class.

This is largely a matter of taste and a question of circumstances, but 
I'd like to point out here that your "natural" is not universal. If you 
take a different approach, namely that a class should encapsulate in 
order to maintain its internal consistency but otherwise be as small as 
possible, then algorithms operating on some data are definitely not part 
of that data. The advantage is that the data class gets smaller, and in 
the algorithms you don't risk ruining the internal integrity of the used 
data.

Further, encapsulating algorithms into classes is also not natural. 
Algorithms are often expressed much better as functions. Shoe-horning 
things into a class in the name of OOP is IMHO misguided.

Concerning mixins, you can put them into separate modules[1]. If it is 
clearly documented that class FooColourMixin handles the colour-related 
stuff for class Foo, and reversely that class Foo inherits FooShapeMixin 
and FooColourMixin that provide handling of shape and colour, then that 
is fine. It allows you to not only encapsulate things inside class Foo 
but to partition things inside Foo. Note that mixins are easier to write 
than in C++. If the mixin needs access to the derived class' function 
bar(), it just calls self.bar(). There is no type-casting or other magic 
involved. The same applies to data attributes (non-function attributes), 
basically all attributes are "virtual". The compile-time, static type 
checking of e.g. C++ doesn't exist.


>> Python lets you inject new methods into existing classes (this is
>> sometimes called duck punching) but I don't recommend doing this.
>
> Is there not a way just to declare the method in the class and put
> the actual implementation in another file on the python path so that
> it's picked up a run time?

To answer your question, no, not directly. Neither is there a separation 
like in C++ between interface and implementation, nor is there something 
like in C# with partial classes. C++ interface/implementation separation 
is roughly provided by abstract base classes. C# partial classes are 
most closely emulated with mixins.

That said, modifying classes is neither magic nor is it uncommon:

   class foo:
       pass

   import algo_x
   foo.algo = algo_x.function

Classes are not immutable, you can add and remove things just like you 
can do with objects.


BTW: If you told us which language(s) you have a background in, it could 
be easier to help you with identifying the idioms in that language that 
turn into misconceptions when applied to Python.

Greetings!

Uli

[1] Actually, modules themselves provide the kind of separation that I 
think you are after. Don't always think "class" if it comes to 
encapsulation and modularization!

[toc] | [prev] | [next] | [standalone]

#32634

> 
> 
> BTW: If you told us which language(s) you have a background in, it could be easier to help you with identifying the idioms in that language that turn into misconceptions when applied to Python.

I'm considering porting some MATLAB code to python to move away from commercial software. Python seemed like the right choice simply because of the wonderful numpy, scipy and matplotlib.

So my project will build on these packages to provide some additional state and functionality.

Cheers,

Martin

> 
> Greetings!
> 
> Uli
> 
> [1] Actually, modules themselves provide the kind of separation that I think you are after. Don't always think "class" if it comes to encapsulation and modularization!
> -- 
> http://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#32638

On 02/11/2012 14:49, Martin Hewitson wrote:

[Top posting fixed]

>
>>
>>
>> BTW: If you told us which language(s) you have a background in, it could be easier to help you with identifying the idioms in that language that turn into misconceptions when applied to Python.
>
>>
>> Greetings!
>>
>> Uli
>>
>> [1] Actually, modules themselves provide the kind of separation that I think you are after. Don't always think "class" if it comes to encapsulation and modularization!
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>
> I'm considering porting some MATLAB code to python to move away from commercial software. Python seemed like the right choice simply because of the wonderful numpy, scipy and matplotlib.
>
> So my project will build on these packages to provide some additional state and functionality.
>
> Cheers,
>
> Martin
>

Just in case you're not aware there are separate mailing lists for numpy 
and matplotlib, presumably scipy as well, should you have specific 
questions.  Further matplotlib is now at version 1.2rc3 with Python 3 
support, yippee.  Combine that with the recently released Python 3.3 and 
it should make one hell of a tool kit :)

-- 
Cheers.

Mark Lawrence.

[toc] | [prev] | [next] | [standalone]

#32678

On Fri, 02 Nov 2012 09:08:07 +0100, Martin Hewitson wrote:

> Even if one takes reasonable numbers: 20 methods, each method has 20
> lines of documentation, then we immediately have 400 lines in the file
> before writing a line of code. It would seem much more natural to me to
> have these methods in their own file, grouped nicely in sub-directories.

Ewww. Ewww ewww ewww ewww. Just reading about it makes me feel dirty.

Seriously. That means any time you want to jump back and forth from one 
method to another method OF THE SAME CLASS, you have to change files. 
Yuck.

-- 
Steven

[toc] | [prev] | [next] | [standalone]

#32602

On 02/11/2012 08:08, Martin Hewitson wrote:
>
> Even if one takes reasonable numbers: 20 methods, each method has 20 lines of documentation, then we immediately have 400 lines in the file before writing a line of code. It would seem much more natural to me to have these methods in their own file, grouped nicely in sub-directories. But it seems this is not the python way. Sigh.
>
> Thanks for your thoughts,
>
> Martin
>

20 lines of documentation per method?  As far as I'm concerned that's 
not a smell, that's a stink.

-- 
Cheers.

Mark Lawrence.

[toc] | [prev] | [next] | [standalone]

#32613

On Fri, 02 Nov 2012 08:40:06 +0000, Mark Lawrence wrote:

> On 02/11/2012 08:08, Martin Hewitson wrote:
>>
>> Even if one takes reasonable numbers: 20 methods, each method has 20
>> lines of documentation, then we immediately have 400 lines in the file
>> before writing a line of code. It would seem much more natural to me to
>> have these methods in their own file, grouped nicely in
>> sub-directories. But it seems this is not the python way. Sigh.
>>
>> Thanks for your thoughts,
>>
>> Martin
>>
>>
> 20 lines of documentation per method?  As far as I'm concerned that's
> not a smell, that's a stink.

Depends on the method. For some, 20 lines is 18 lines too many. For 
others, that's 80 lines too few.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#32607

On Fri, Nov 2, 2012 at 7:08 PM, Martin Hewitson <martinhewitson@mac.com> wrote:
>
> On 2, Nov, 2012, at 08:38 AM, Paul Rubin <no.email@nospam.invalid> wrote:
>
>> Martin Hewitson <martinhewitson@mac.com> writes:
>>> So, is there a way to put these methods in their own files and have
>>> them 'included' in the class somehow? ... Is there an official python
>>> way to do this? I don't like having source files with 100's of lines
>>> of code in, let alone 1000's.
>>
>> That code sounds kind of smelly... why are there so many methods per
>> class?
>
> Simply because there are many different ways to process the data. The class encapsulates the data, and the user can process the data in many ways. Of course, one could have classes which encapsulate the algorithms, as well as the data, but it also seems natural to me to have algorithms as methods which are part of the data class, so the user operates on the data using methods of that class.

Are these really needing to be methods, or ought they to be
module-level functions? Remember, Python code doesn't have to be
organized into classes the way Java code is.

ChrisA

[toc] | [prev] | [next] | [standalone]

#32610

On 2, Nov, 2012, at 09:40 AM, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:

> On 02/11/2012 08:08, Martin Hewitson wrote:
>> 
>> Even if one takes reasonable numbers: 20 methods, each method has 20 lines of documentation, then we immediately have 400 lines in the file before writing a line of code. It would seem much more natural to me to have these methods in their own file, grouped nicely in sub-directories. But it seems this is not the python way. Sigh.
>> 
>> Thanks for your thoughts,
>> 
>> Martin
>> 
> 
> 20 lines of documentation per method?  As far as I'm concerned that's not a smell, that's a stink.

Wow, I don't think I've ever been criticised before for writing too much documentation :)

I guess we have different end users. This is not a set of classes for other developers to use: it's a set of classes which creates a data analysis environment for scientists to use. They are not programmers, and expect the algorithms to be documented in detail.

Martin

> 
> -- 
> Cheers.
> 
> Mark Lawrence.
> 
> -- 
> http://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#32614

Martin Hewitson wrote:

> On 2, Nov, 2012, at 09:40 AM, Mark Lawrence <breamoreboy@yahoo.co.uk>
> wrote:

>> 20 lines of documentation per method?  As far as I'm concerned that's not
>> a smell, that's a stink.
> 
> Wow, I don't think I've ever been criticised before for writing too much
> documentation :)
> 
> I guess we have different end users. This is not a set of classes for
> other developers to use: it's a set of classes which creates a data
> analysis environment for scientists to use. They are not programmers, and
> expect the algorithms to be documented in detail.

While I would never discourage thorough documentation you may be better off 
with smaller docstrings and the details in an external document. Python 
projects typically use rst-files processed by sphinx.

http://pypi.python.org/pypi/Sphinx/

[toc] | [prev] | [next] | [standalone]

#32616

On 02/11/2012 08:45, Martin Hewitson wrote:
>
> On 2, Nov, 2012, at 09:40 AM, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
>
>> On 02/11/2012 08:08, Martin Hewitson wrote:
>>>
>>> Even if one takes reasonable numbers: 20 methods, each method has 20 lines of documentation, then we immediately have 400 lines in the file before writing a line of code. It would seem much more natural to me to have these methods in their own file, grouped nicely in sub-directories. But it seems this is not the python way. Sigh.
>>>
>>> Thanks for your thoughts,
>>>
>>> Martin
>>>
>>
>> 20 lines of documentation per method?  As far as I'm concerned that's not a smell, that's a stink.
>
> Wow, I don't think I've ever been criticised before for writing too much documentation :)
>
> I guess we have different end users. This is not a set of classes for other developers to use: it's a set of classes which creates a data analysis environment for scientists to use. They are not programmers, and expect the algorithms to be documented in detail.
>
> Martin
>

You've completely missed the point.  99% of the time if you can't write 
down what a method does in at most half a dozen lines, the method is 
screaming out to be refactored.  Rightly or wrongly you've already 
rejected that option, although I suspect that rightly is nearer the mark 
in this case on the grounds that practicality beats purity.

-- 
Cheers.

Mark Lawrence.

[toc] | [prev] | [next] | [standalone]

#32618

On 11/2/12 10:21 AM, Peter Otten wrote:
> Martin Hewitson wrote:
>
>> On 2, Nov, 2012, at 09:40 AM, Mark Lawrence <breamoreboy@yahoo.co.uk>
>> wrote:
>
>>> 20 lines of documentation per method?  As far as I'm concerned that's not
>>> a smell, that's a stink.
>>
>> Wow, I don't think I've ever been criticised before for writing too much
>> documentation :)
>>
>> I guess we have different end users. This is not a set of classes for
>> other developers to use: it's a set of classes which creates a data
>> analysis environment for scientists to use. They are not programmers, and
>> expect the algorithms to be documented in detail.
>
> While I would never discourage thorough documentation you may be better off
> with smaller docstrings and the details in an external document. Python
> projects typically use rst-files processed by sphinx.
>
> http://pypi.python.org/pypi/Sphinx/

In the science/math community, we tend to build the Sphinx API reference from 
the thorough, authoritative docstrings. We like having complete docstrings 
because we are frequently at the interactive prompt. We tend to have broad APIs, 
so having a single source of documentation and not repeating ourselves is important.

   http://docs.scipy.org/doc/numpy/reference/index.html
   http://docs.scipy.org/doc/scipy/reference/index.html
   http://www.sagemath.org/doc/reference/
   http://docs.sympy.org/0.7.2/modules/index.html
   http://scikit-learn.org/stable/modules/classes.html

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#32619

On 11/2/12 10:48 AM, Mark Lawrence wrote:
> On 02/11/2012 08:45, Martin Hewitson wrote:
>>
>> On 2, Nov, 2012, at 09:40 AM, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
>>
>>> On 02/11/2012 08:08, Martin Hewitson wrote:
>>>>
>>>> Even if one takes reasonable numbers: 20 methods, each method has 20 lines
>>>> of documentation, then we immediately have 400 lines in the file before
>>>> writing a line of code. It would seem much more natural to me to have these
>>>> methods in their own file, grouped nicely in sub-directories. But it seems
>>>> this is not the python way. Sigh.
>>>>
>>>> Thanks for your thoughts,
>>>>
>>>> Martin
>>>>
>>>
>>> 20 lines of documentation per method?  As far as I'm concerned that's not a
>>> smell, that's a stink.
>>
>> Wow, I don't think I've ever been criticised before for writing too much
>> documentation :)
>>
>> I guess we have different end users. This is not a set of classes for other
>> developers to use: it's a set of classes which creates a data analysis
>> environment for scientists to use. They are not programmers, and expect the
>> algorithms to be documented in detail.
>>
>> Martin
>
> You've completely missed the point.  99% of the time if you can't write down
> what a method does in at most half a dozen lines, the method is screaming out to
> be refactored.  Rightly or wrongly you've already rejected that option, although
> I suspect that rightly is nearer the mark in this case on the grounds that
> practicality beats purity.

You've completely missed the context. These are not really complicated methods 
doing lots of things all at once such that can be refactored to simpler methods. 
The docstrings are not just glorified comments for other developers reading the 
source code. They are the online documentation for non-programmer end-users who 
are using the interactive prompt as an interactive data analysis environment. 
Frequently, they not only have to describe what it's doing, but also introduce 
the whole concept of what it's doing, why you would want to do such a thing, and 
provide examples of its use. That's why they are so long. For example:

http://docs.scipy.org/doc/numpy/reference/generated/numpy.fft.fft.html

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#32676

On Fri, 02 Nov 2012 07:16:09 +0100, Martin Hewitson wrote:

> I'm beginning a large Python project which contains many packages,
> modules and classes. The organisation of those is clear to me.
[...]
> I don't like having source files with
> 100's of lines of code in, let alone 1000's.

Why? Do you do your software development on an iPhone?

Hundreds of lines is nothing, especially if you are including docstrings 
and comments in that count. My Python startup file is over 100 lines, and 
it is trivial.

100 lines is approximately and a half pages using a 10pt font size 
(depending on the font and the platform, of course). In any case, it's 
not a lot. If you average two lines of code every comment, blank line or 
docstring, 100 lines is only 66 lines of actual code.

To give you an example of what I consider pushing the limit of how much 
code should go into a single file, look at the source code to decimal.py. 
In Python 3.2, that file is 6249 lines. There are:

- 2880 lines of actual code
- 2070 lines in docstrings, including blanks
- 606 commented lines
- 693 blank lines outside of docstrings

The module includes 20 top-level functions, 18 classes, and 213 methods. 
Most of those methods are in just two classes, Context and Decimal, with 
76 and 117 methods respectively. So there is an average of 12 lines of 
code per function or method.

I wouldn't like to have to deal with a single file twice that size, but 
decimal.py is relatively easy to deal with. It's not a trivial module by 
any means, but nor is it especially over-complex. The complexity is all 
in the computations, not the code layout.

Please understand me -- I'm not saying that you should stuff all your 
code into a single module. Code should only be placed into a single 
module when the code is related. But in my opinion, you should not split 
a logical single unit of code, a module of related code, into separate 
files just because it is "too big" until it is at least as large as 
decimal.py.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#32677

On Sat, 03 Nov 2012 01:06:45 +0000, Steven D'Aprano wrote:

> 100 lines is approximately and a half pages using a 10pt font size
> (depending on the font and the platform, of course).

Crap. I meant approx *one* and a half pages.



-- 
Steven

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web