Python 2 to 3 conversion - embrace the pain

From	John Nagle <nagle@animats.com>
Newsgroups	comp.lang.python
Subject	Python 2 to 3 conversion - embrace the pain
Date	2015-03-13 14:08 -0700
Organization	A noiseless patient Spider
Message-ID	<550351BF.5020108@animats.com> (permalink)

Show all headers | View raw

  I'm approaching the end of converting a large system from Python 2 to
Python 3.  Here's why you don't want to do this.

  The language changes aren't that bad, and they're known and
documented.  It's the package changes that are the problem.
Discovering and fixing all the new bugs takes a while.


BeautifulSoup:

BeautifulSoup 3 has been phased out. I had my own version of
BeautifulSoup 3, modified for greater robustness.  But that was
years ago.  So I converted to BeautifulSoup 4, as the documentation
says to do.

The HTML5parser module is claimed to parse as a browser does, with
all the error tolerance specified in the HTML5 spec. (The spec
actually specifies how to handle bad HTML consistently across
browsers in great detail, and HTML5parser has code in it for that.)

It doesn't deliver on that promise, though. Some sites crash
BeautifulSoup 4/HTML5parser.  Try "kroger.com", which has HTML with
<head><head>.  The parse tree constructed has a bad link,
and trying to use the parse tree results in exceptions.
Submitted bug report.  Appears to be another case of
a known bug.  No workaround at this time.

https://bugs.launchpad.net/beautifulsoup/+bug/1270611
https://bugs.launchpad.net/beautifulsoup/+bug/1430633


PyMySQL:

"Pymysql is a pure Python drop-in replacement for MySQLdb".
Sounds good.  Then I discover that LOAD DATA LOCAL wasn't
implemented in the version on PyPi.  It's on Github, though,
and I got the authors to push that out to PyPi.  It
works on test cases.  But it doesn't work on a big job,
because the default size of MySQL packets was set to 16MB.
This made the LOAD DATA LOCAL code try to send the entire
file being loaded as one giant MySQL packet.  Unless you
configure the MySQL server with 16MB buffers, this fails, with
an obscure "server has gone away" message.  Found the
problem, came up with a workaround, submitted a bug report,
and it's being fixed.

https://github.com/PyMySQL/PyMySQL/issues/317


SSL:

All the new TLS/SSL support is in Python 3. That's good.
Unfortunately, using Firefox's set of SSL certs, some
important sites (such as "verisign.com") don't validate.
This turned out to be a complex problem involving Verisign
cross-signing a certificate, which created a certificate
hierarchy that some versions of OpenSSL can't handle.
There's now a version of OpenSSL that can handle it, but
the Python library has to make a call to use it, and
that's going in but isn't deployed yet.  This bug
resulted in much finger-pointing between the Python
and OpenSSL developers, the Mozilla certificate store
maintainers, and Verisign.  It's now been sorted out,
but not all the fixes are deployed.  Because "ssl" is
a core Python module, this will remain broken until the
next Python release, on both the 2.7 and 3.4 lines.

Also, for no particularly good reason, the exception
"SSL.CertificateError" is not a subclass of "SSL.Error",
resulting in a routine exception not being recognized.

Bug reports submitted for both OpenSSL and Python SSL.
Much discussion.  Problem fixed, but fix is in next
version of Python.  No workaround at this time.

http://bugs.python.org/issue23476


Pickle:

As I just posted recently, CPickle on Python 3.4 seems to
have a memory corruption bug.  Pure-Python Pickle is fine.
So a workaround is possible.  Bug report submitted.

http://bugs.python.org/issue23655


Converting a large application program to Python 3
thus required diagnosing four library bugs and filing
bug reports on all of them.  Workarounds are known
for two of the problems.  I can't deploy the Python 3
version on the servers yet.

				John Nagle

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Python 2 to 3 conversion - embrace the pain John Nagle <nagle@animats.com> - 2015-03-13 14:08 -0700
  Re: Python 2 to 3 conversion - embrace the pain INADA Naoki <songofacandy@gmail.com> - 2015-03-14 07:27 +0900
    Re: Python 2 to 3 conversion - embrace the pain John Nagle <nagle@animats.com> - 2015-03-13 16:01 -0700
      Re: Python 2 to 3 conversion - embrace the pain Ned Deily <nad@acm.org> - 2015-03-13 16:14 -0700
      Re: Python 2 to 3 conversion - embrace the pain Chris Angelico <rosuav@gmail.com> - 2015-03-14 12:40 +1100
      Re: Python 2 to 3 conversion - embrace the pain INADA Naoki <songofacandy@gmail.com> - 2015-03-15 16:01 +0900
        Re: Python 2 to 3 conversion - embrace the pain wxjmfauth@gmail.com - 2015-03-15 01:11 -0700
  Re: Python 2 to 3 conversion - embrace the pain Marko Rauhamaa <marko@pacujo.net> - 2015-03-14 10:00 +0200
    Re: Python 2 to 3 conversion - embrace the pain John Nagle <nagle@animats.com> - 2015-03-15 12:05 -0700
      Re: Python 2 to 3 conversion - embrace the pain Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-15 19:16 +0000
      Re: Python 2 to 3 conversion - embrace the pain "Fetchinson ." <fetchinson@googlemail.com> - 2015-03-15 21:59 +0100
      Re: Python 2 to 3 conversion - embrace the pain Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-15 21:05 +0000
        Re: Python 2 to 3 conversion - embrace the pain Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-16 11:25 +1100
          Re: Python 2 to 3 conversion - embrace the pain Chris Angelico <rosuav@gmail.com> - 2015-03-16 11:38 +1100
          Re: Python 2 to 3 conversion - embrace the pain Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-16 01:07 +0000
      Re: Python 2 to 3 conversion - embrace the pain Mario Figueiredo <marfig@gmail.com> - 2015-03-16 00:23 +0100
        Re: Python 2 to 3 conversion - embrace the pain Roy Smith <roy@panix.com> - 2015-03-15 19:43 -0400
          Re: Python 2 to 3 conversion - embrace the pain Chris Angelico <rosuav@gmail.com> - 2015-03-16 11:00 +1100
          Re: Python 2 to 3 conversion - embrace the pain Mario Figueiredo <marfig@gmail.com> - 2015-03-16 01:19 +0100
          Re: Python 2 to 3 conversion - embrace the pain John Nagle <nagle@animats.com> - 2015-03-17 23:37 -0700
            Re: Python 2 to 3 conversion - embrace the pain Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-18 08:41 +0000
            Re: Python 2 to 3 conversion - embrace the pain Ned Deily <nad@acm.org> - 2015-03-18 02:11 -0700
              Re: Python 2 to 3 conversion - embrace the pain Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-18 21:53 +1100
                Re: Python 2 to 3 conversion - embrace the pain Rustom Mody <rustompmody@gmail.com> - 2015-03-18 06:42 -0700
                Re: Python 2 to 3 conversion - embrace the pain wxjmfauth@gmail.com - 2015-03-18 07:23 -0700
                Re: Python 2 to 3 conversion - embrace the pain Paul Rubin <no.email@nospam.invalid> - 2015-03-18 09:35 -0700
                Re: Python 2 to 3 conversion - embrace the pain Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-20 09:48 +1100
        Re: Python 2 to 3 conversion - embrace the pain Kishan Thobhani <thobhanikishan@gmail.com> - 2015-03-16 05:05 +0530
        Re: Python 2 to 3 conversion - embrace the pain Kishan Thobhani <thobhanikishan@gmail.com> - 2015-03-16 05:23 +0530
  Re: Python 2 to 3 conversion - embrace the pain Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-14 09:05 +0000

csiph-web