Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #57406 > unrolled thread

Maintaining a backported module

Back to article view | Back to comp.lang.python

Contents

  Maintaining a backported module Steven D'Aprano <steve@pearwood.info> - 2013-10-24 04:54 +0000
    Re: Maintaining a backported module Ben Finney <ben+python@benfinney.id.au> - 2013-10-24 16:09 +1100
      Re: Maintaining a backported module Metallicow <metaliobovinus@gmail.com> - 2013-10-23 22:46 -0700
        Re: Maintaining a backported module Metallicow <metaliobovinus@gmail.com> - 2013-10-30 03:32 -0700
    Re: Maintaining a backported module Paul Rubin <no.email@nospam.invalid> - 2013-10-23 22:38 -0700
    Re: Maintaining a backported module Ethan Furman <ethan@stoneleaf.us> - 2013-10-23 23:30 -0700
    Re: Maintaining a backported module Ben Finney <ben+python@benfinney.id.au> - 2013-10-24 17:59 +1100
      Re: Maintaining a backported module Steven D'Aprano <steve@pearwood.info> - 2013-10-24 09:38 +0000
        Re: Maintaining a backported module Ben Finney <ben+python@benfinney.id.au> - 2013-10-24 21:01 +1100
    Re: Maintaining a backported module Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2013-10-24 09:00 +0200
    Re: Maintaining a backported module Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-24 09:37 +0100
    Re: Maintaining a backported module Ned Batchelder <ned@nedbatchelder.com> - 2013-10-24 06:36 -0400
      Re: Maintaining a backported module Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-10-24 11:46 +0000
        Re: Maintaining a backported module rusi <rustompmody@gmail.com> - 2013-10-24 05:25 -0700
        Re: Maintaining a backported module Ned Batchelder <ned@nedbatchelder.com> - 2013-10-24 13:46 -0400
        Re: Maintaining a backported module Terry Reedy <tjreedy@udel.edu> - 2013-10-24 16:37 -0400
        Python Coverage: testing a program (was: Maintaining a backported module) Ben Finney <ben+python@benfinney.id.au> - 2013-10-25 07:54 +1100
        Re: Python Coverage: testing a program Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-24 22:55 +0100
        Re: Python Coverage: testing a program Terry Reedy <tjreedy@udel.edu> - 2013-10-24 18:36 -0400
        Re: Python Coverage: testing a program Ethan Furman <ethan@stoneleaf.us> - 2013-10-24 15:28 -0700
        Re: Python Coverage: testing a program Ben Finney <ben+python@benfinney.id.au> - 2013-10-25 10:27 +1100
        Re: Python Coverage: testing a program Ben Finney <ben+python@benfinney.id.au> - 2013-10-25 10:30 +1100
        Re: Python Coverage: testing a program Ned Batchelder <ned@nedbatchelder.com> - 2013-10-24 19:53 -0400
        Re: Python Coverage: testing a program Terry Reedy <tjreedy@udel.edu> - 2013-10-24 19:55 -0400
        Re: Maintaining a backported module Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-10-25 08:49 +0100
    Re: Maintaining a backported module Christian Heimes <christian@python.org> - 2013-10-24 12:47 +0200
    Re: Maintaining a backported module Ethan Furman <ethan@stoneleaf.us> - 2013-10-24 12:42 -0700

Page 1 of 2  [1] 2  Next page →

#57406 — Maintaining a backported module

As some of you are aware, I have a module accepted into the standard 
library:

http://docs.python.org/3.4/library/statistics.html

I'm now at the point where I wish to backport this module to support 
versions of Python back to 3.1 at least and possibly 2.7, and put it up 
on PyPI.

I'm looking for advice on best practices for doing so. Any suggestions 
for managing bug fixes and enhancements to two separate code-bases 
without them diverging too much?

Other than "Avoid it" :-)



-- 
Steven

[toc] | [next] | [standalone]

#57409

Steven D'Aprano <steve@pearwood.info> writes:

> I'm now at the point where I wish to backport this module to support
> versions of Python back to 3.1 at least and possibly 2.7, and put it
> up on PyPI.
>
> I'm looking for advice on best practices for doing so. Any suggestions 
> for managing bug fixes and enhancements to two separate code-bases 
> without them diverging too much?

It is a great advantage that you've targeted Python 3 primarily. As I
understand it, it is far easier to make an existing Python 3 code base
also work on Python 2, than vice versa.

(Though that will obviously change as Python 3 continues to diverge from
Python 2. The more one depends on Python 3 features, the more one
depends on back-ported versions of those features for Python 2.

I'm pretty sure that's not going to be much of an issue for the
‘statistics’ package, but it's worth mentioning as a general caveat.)

> Other than "Avoid it" :-)

A useful library for this purpose is ‘six’ (as in “3 × 2”)
<URL:http://pythonhosted.org/six/>. You can use its features to do
things that are useful or better in Python 3, but which need special
implementation to work on Python 2; and the same code will just work on
both versions.

-- 
 \         “Pinky, are you pondering what I'm pondering?” “I think so, |
  `\    Brain, but if we get Sam Spade, we'll never have any puppies.” |
_o__)                                           —_Pinky and The Brain_ |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#57414

On Thursday, October 24, 2013 12:09:55 AM UTC-5, Ben Finney wrote:
> A useful library for this purpose is ‘six’ (as in “3 × 2”)
> <URL:http://pythonhosted.org/six/>. You can use its features to do
> things that are useful or better in Python 3, but which need special
> implementation to work on Python 2; and the same code will just work on
> both versions.

+1 for stdev Steven. Thanks for the extra legs.
Hope all goes well with introductions... I'm sure it will.
:) Good Job.

[toc] | [prev] | [next] | [standalone]

#58025

On Thursday, October 24, 2013 12:46:51 AM UTC-5, Metallicow wrote:
> +1 for stdev Steven. Thanks for the extra legs.
> Hope all goes well with introductions... I'm sure it will.
> :) Good Job.

Well, what I am trying to get at is whether it is better as...

stddev or stdev...? 6(3standard abc) vs 5(2nonstandard abc)
How many people are going to have to change their code...

Is alphabetsoup ABC better than oppsImissedAspoonfull(AC)?

As far as readability and standard naming conventions are concerned?

[toc] | [prev] | [next] | [standalone]

#57411

Steven D'Aprano <steve@pearwood.info> writes:
> As some of you are aware, I have a module accepted into the standard 
> library:
> http://docs.python.org/3.4/library/statistics.html

Wow, neat, I had seen something about the module and thought it looked
great, but I didn't realize you were the author.  Awesome!

> Any suggestions for managing bug fixes and enhancements to two
> separate code-bases without them diverging too much?

I haven't touched 3.x yet, but I found with a little bit of care in
resisting use of new features, I was able to implement in the
intersection of 2.x and 1.5, so the same code worked in both.  That
avoided the need for two codebases in the stuff I did.  I don't know if
it would be more difficult with the stats module.

[toc] | [prev] | [next] | [standalone]

#57418

On 10/23/2013 09:54 PM, Steven D'Aprano wrote:
>
> I'm looking for advice on best practices for doing so. Any suggestions
> for managing bug fixes and enhancements to two separate code-bases
> without them diverging too much?

Confining your code to the intersection of 2.7 and 3.x is probably going to be the easiest thing to do as 2.7 has a 
bunch of 3.x features.

Sadly, when I backported Enum I was targeting 2.5 - 3.x because I have systems still running 2.5.  That was *not* a fun 
experience.  :(

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

#57419

Steven D'Aprano <steve@pearwood.info> writes:

> I'm now at the point where I wish to backport this module to support
> versions of Python back to 3.1 at least and possibly 2.7, and put it
> up on PyPI.

Ned Batchelder has managed something at least as ambitious (supporting
Python versions 2.4 through 3.3), which should be helpful in your case
<URL:http://nedbatchelder.com/blog/200910/running_the_same_code_on_python_2x_and_3x.html>.

Eli Bendersky also has an article with specific advice
<URL:http://eli.thegreenplace.net/2010/05/19/making-code-compatible-with-python-2-and-3/>.

-- 
 \       “If you are unable to leave your room, expose yourself in the |
  `\            window.” —instructions in case of fire, hotel, Finland |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#57441

On Thu, 24 Oct 2013 17:59:59 +1100, Ben Finney wrote:

> Steven D'Aprano <steve@pearwood.info> writes:
> 
>> I'm now at the point where I wish to backport this module to support
>> versions of Python back to 3.1 at least and possibly 2.7, and put it up
>> on PyPI.
> 
> Ned Batchelder has managed something at least as ambitious (supporting
> Python versions 2.4 through 3.3), which should be helpful in your case
> <URL:http://nedbatchelder.com/blog/200910/
running_the_same_code_on_python_2x_and_3x.html>.

I too have written code that supports 2.4 - 3.3 (although only relatively 
small modules). That's not the problem here. My problem is that I have 
two chunks of code:

1) statistics.py in the standard library, which is written for 
   Python 3.3/3.4 only, and should be as clean as possible;

2) a backport of it, on PyPI, which will support older Pythons, and
   may be slower/uglier if need be.

My problem is not supporting 2.7 and 3.4 in the one code base. My problem 
is, how do I prevent #1 and #2 from gradually diverging?

The easy answer is "unit tests", but the unit tests for 1) are in the std 
lib and target 3.4, while the unit tests for 2) will be on PyPI and 
won't. So how do I keep the unit tests from diverging?



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#57442

Steven D'Aprano <steve@pearwood.info> writes:

> 1) statistics.py in the standard library, which is written for 
>    Python 3.3/3.4 only, and should be as clean as possible;
>
> 2) a backport of it, on PyPI, which will support older Pythons, and
>    may be slower/uglier if need be.
>
> My problem is not supporting 2.7 and 3.4 in the one code base. My
> problem is, how do I prevent #1 and #2 from gradually diverging?

Generate one of them automatically from the other. My recommendation
would be to edit only the code written for Python 3.4, have a
fully-automatic generator for the code targeting Python 2.7, and ignore
all earlier versions.

How does the “3to2” tool <URL:http://pypi.python.org/pypi/3to2/> fare
with converting your code?

> The easy answer is "unit tests", but the unit tests for 1) are in the
> std lib and target 3.4, while the unit tests for 2) will be on PyPI
> and won't. So how do I keep the unit tests from diverging?

Again, I'd recommend you generate the Python 2 code automatically from
the actively-maintained Python 3 code.

-- 
 \     “Beware of and eschew pompous prolixity.” —Charles A. Beardsley |
  `\                                                                   |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#57421

Op 24-10-13 06:54, Steven D'Aprano schreef:
> As some of you are aware, I have a module accepted into the standard 
> library:
> 
> http://docs.python.org/3.4/library/statistics.html
> 
> I'm now at the point where I wish to backport this module to support 
> versions of Python back to 3.1 at least and possibly 2.7, and put it up 
> on PyPI.
> 
> I'm looking for advice on best practices for doing so. Any suggestions 
> for managing bug fixes and enhancements to two separate code-bases 
> without them diverging too much?
> 
> Other than "Avoid it" :-)

I would use only one code-base and make a branch for the 2.7 version.
You can then adapt the 2.7 version where really necessary but in general
just merge adaption made in the main branch to the 2.7 branch.

I am assuming you are using some kind of version control.

-- 
Antoon Pardon

[toc] | [prev] | [next] | [standalone]

#57435

On 24/10/2013 07:30, Ethan Furman wrote:
> On 10/23/2013 09:54 PM, Steven D'Aprano wrote:
>>
>> I'm looking for advice on best practices for doing so. Any suggestions
>> for managing bug fixes and enhancements to two separate code-bases
>> without them diverging too much?
>
> Confining your code to the intersection of 2.7 and 3.x is probably going
> to be the easiest thing to do as 2.7 has a bunch of 3.x features.
>
> Sadly, when I backported Enum I was targeting 2.5 - 3.x because I have
> systems still running 2.5.  That was *not* a fun experience.  :(
>
> --
> ~Ethan~

Have you or could you publish anything regarding your experiences, I 
suspect it would be an enlightening read for a lot of us?

-- 
Python is the second best programming language in the world.
But the best has yet to be invented.  Christian Tismer

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#57445

On 10/24/13 2:59 AM, Ben Finney wrote:
> Steven D'Aprano <steve@pearwood.info> writes:
>
>> I'm now at the point where I wish to backport this module to support
>> versions of Python back to 3.1 at least and possibly 2.7, and put it
>> up on PyPI.
> Ned Batchelder has managed something at least as ambitious (supporting
> Python versions 2.4 through 3.3), which should be helpful in your case
> <URL:http://nedbatchelder.com/blog/200910/running_the_same_code_on_python_2x_and_3x.html>.

FWIW, coverage.py currently runs on 2.3 through 3.4.  It mostly comes 
down to:

1. avoiding newer features (decorators! generator expressions! rpartition!),
2. using a compatibility layer like "six" (though I started my own 
called backward.py before six existed),
3. using some awkward syntax workarounds (sys.exc_info()[1] to get the 
current exception),
4. somehow finding a way to test on all those versions (pythonz helps, 
and you have to limit your dependencies).

Also, I've just started the coverage.py 4.x branch, which will run on 
 >=2.6 and >=3.2, and it's very nice to get rid of some of that 
compatibility stuff.

--Ned.
> Eli Bendersky also has an article with specific advice
> <URL:http://eli.thegreenplace.net/2010/05/19/making-code-compatible-with-python-2-and-3/>.
>

[toc] | [prev] | [next] | [standalone]

#57448

On Thu, 24 Oct 2013 06:36:04 -0400, Ned Batchelder wrote:

> coverage.py currently runs on 2.3 through 3.4

You support all the way back to 2.3???

I don't know whether to admire your dedication, or back away slowly since 
you're obviously a crazy person :-)



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#57452

On Thursday, October 24, 2013 5:16:58 PM UTC+5:30, Steven D'Aprano wrote:
> On Thu, 24 Oct 2013 06:36:04 -0400, Ned Batchelder wrote:
> 
> > coverage.py currently runs on 2.3 through 3.4
> 
> You support all the way back to 2.3???
> 
> I don't know whether to admire your dedication, or back away slowly since 
> you're obviously a crazy person :-)

Yes, in the end youve to choose between a rock and a hard place:
Either you write idiomatic, beautiful 3-code that is not 2-compatible or you write not so beautiful code putting compatibility at the highest priority.

Cant think of examples right now but Ive seen very old python code with  True, False defined and inside try-block and all that.  Looks ugly but is compatibility-wise correct adn appropriate

BTW saw this yesterday: 
An interesting read correlating OSS projects that succeed and fail with the difference between a commitment to computer science and to computer engineering.
Consider it an early-warning system!

http://www.forbes.com/sites/danwoods/2013/10/22/why-hasnt-open-source-taken-over-storage/

[toc] | [prev] | [next] | [standalone]

#57463

On 10/24/13 7:46 AM, Steven D'Aprano wrote:
> On Thu, 24 Oct 2013 06:36:04 -0400, Ned Batchelder wrote:
>
>> coverage.py currently runs on 2.3 through 3.4
> You support all the way back to 2.3???
>
> I don't know whether to admire your dedication, or back away slowly since
> you're obviously a crazy person :-)
>
>
>

Yeah, it's kind of crazy.  It was a boiling frog situation: the package 
supported 2.3 back when that was just normal, and there was no obvious 
time to drop it, so it's been carried along.

It's been fun dropping the contortions for coverage.py 4.x, though!

--Ned.

[toc] | [prev] | [next] | [standalone]

#57468

On 10/24/2013 1:46 PM, Ned Batchelder wrote:
>> On Thu, 24 Oct 2013 06:36:04 -0400, Ned Batchelder wrote:
>>> coverage.py currently runs on 2.3 through 3.4

I want to thank you for this package. I have used it when writing test 
modules for idlelib modules and aiming for 100% coverage forces me to 
really understand the code tested to hit all the conditional lines.

> It's been fun dropping the contortions for coverage.py 4.x, though!

One request: ignore "if __name__ == '__main__':" clauses at the end of 
files, which cannot be run under coverage.py, so 100% coverage is 
reported as 100% instead of 9x%.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#57470 — Python Coverage: testing a program (was: Maintaining a backported module)

Terry Reedy <tjreedy@udel.edu> writes:

> On 10/24/2013 1:46 PM, Ned Batchelder wrote:
> > It's been fun dropping the contortions for coverage.py 4.x, though!
>
> One request: ignore "if __name__ == '__main__':" clauses at the end of
> files, which cannot be run under coverage.py, so 100% coverage is
> reported as 100% instead of 9x%.

You can do this already with current Coverage: tell Coverage to exclude
<URL:http://nedbatchelder.com/code/coverage/excluding.html> specific
statements, and it won't count them for coverage calculations.

-- 
 \         “If we don't believe in freedom of expression for people we |
  `\           despise, we don't believe in it at all.” —Noam Chomsky, |
_o__)                                                       1992-11-25 |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#57475 — Re: Python Coverage: testing a program

On 24/10/2013 21:54, Ben Finney wrote:
> Terry Reedy <tjreedy@udel.edu> writes:
>
>> On 10/24/2013 1:46 PM, Ned Batchelder wrote:
>>> It's been fun dropping the contortions for coverage.py 4.x, though!
>>
>> One request: ignore "if __name__ == '__main__':" clauses at the end of
>> files, which cannot be run under coverage.py, so 100% coverage is
>> reported as 100% instead of 9x%.
>
> You can do this already with current Coverage: tell Coverage to exclude
> <URL:http://nedbatchelder.com/code/coverage/excluding.html> specific
> statements, and it won't count them for coverage calculations.
>

An alternative to Ned's great bit of kit is figleaf.  This, Ned's and 
many other useful tools are listed here 
https://wiki.python.org/moin/PythonTestingToolsTaxonomy

-- 
Python is the second best programming language in the world.
But the best has yet to be invented.  Christian Tismer

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#57477 — Re: Python Coverage: testing a program

On 10/24/2013 4:54 PM, Ben Finney wrote:
> Terry Reedy <tjreedy@udel.edu> writes:
>
>> On 10/24/2013 1:46 PM, Ned Batchelder wrote:
>>> It's been fun dropping the contortions for coverage.py 4.x, though!
>>
>> One request: ignore "if __name__ == '__main__':" clauses at the end of
>> files, which cannot be run under coverage.py, so 100% coverage is
>> reported as 100% instead of 9x%.
>
> You can do this already with current Coverage: tell Coverage to exclude
> <URL:http://nedbatchelder.com/code/coverage/excluding.html> specific
> statements, and it won't count them for coverage calculations.

OK, I added .coveragerc and that works. In the process of verifying 
this, I was reminded that there is an overt bug in the html report as 
displayed by Firefox. The fonts used for line numbers 
("class='linenos'") and line text ("class='text'") are slightly 
different and, more importantly, with difference sizes (line numbers are 
larger). So corresponding numbers and text do not line up with each 
other. This makes the J,K hotkeys and missing branch notations much less 
useful than intended. If I use cntl-scrollwheel to change text size, 
both change in the same proportion, so the mismatch is maintained.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#57478 — Re: Python Coverage: testing a program

On 10/24/2013 01:54 PM, Ben Finney wrote:
> Terry Reedy <tjreedy@udel.edu> writes:
>
>> On 10/24/2013 1:46 PM, Ned Batchelder wrote:
>>> It's been fun dropping the contortions for coverage.py 4.x, though!
>>
>> One request: ignore "if __name__ == '__main__':" clauses at the end of
>> files, which cannot be run under coverage.py, so 100% coverage is
>> reported as 100% instead of 9x%.
>
> You can do this already with current Coverage: tell Coverage to exclude
> <URL:http://nedbatchelder.com/code/coverage/excluding.html> specific
> statements, and it won't count them for coverage calculations.

While that's neat (being able to exclude items) is there any reason to 
ever count the `if __name__ == '__main__'` clause?  Are there any 
circumstances where it could run under Coverage?  (Apologies if this is 
a dumb question, I know nothing about Coverage myself -- but I'm going 
to go look it up now.  ;)

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web