Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #43522 > unrolled thread

API design for Python 2 / 3 compatibility

Back to article view | Back to comp.lang.python

Contents

  API design for Python 2 / 3 compatibility Stefan Schwarzer <sschwarzer@sschwarzer.net> - 2013-04-13 18:36 +0200
    Re: API design for Python 2 / 3 compatibility Terry Jan Reedy <tjreedy@udel.edu> - 2013-04-13 13:32 -0400
    Re: API design for Python 2 / 3 compatibility Ethan Furman <ethan@stoneleaf.us> - 2013-04-13 11:03 -0700
    Re: API design for Python 2 / 3 compatibility Stefan Schwarzer <sschwarzer@sschwarzer.net> - 2013-04-14 13:11 +0200

#43522 — API design for Python 2 / 3 compatibility

Hello,

I'm currently changing the FTP client library ftputil [1]
so that the same code of the library works with Python
2 (2.6 and up) and Python 3. (At the moment the code is for
Python 2 only.) I've run into a API design issue where I
don't know which API I should offer ftputil users under
Python 2 and Python 3.

[1] http://ftputil.sschwarzer.net/

Some important background information: A key idea in ftputil
is that it uses the same APIs as the Python standard library
where possible. For example, with ftputil you can write code
like this:

  with ftputil.FTPHost(host, user, password) as ftp_host:
      # Like `os.path.isdir`, but works on the FTP server.
      if ftp_host.path.isdir("hello_dir"):
          # Like `os.chdir`, but works on the FTP server.
          ftp_host.chdir("hello_dir")
          # Like the `open` builtin, but opens a remote file.
          with ftp_host.open("new_file", "w") as fobj:
              # Like `file.write` and `file.close`
              fobj.write("Hello world!")
              fobj.close()

Since most of Python 2's and Python 3's filesystem-related
APIs accept either bytes and character strings (and return
the same type if they return anything string-like at all),
the design here is rather clear to me.

However, I have some difficulty with ftputil's counterpart
of the `open` builtin function when files are opened for
reading in text mode. Here are the approaches I've been
thinking of so far:

* Approach 1

  When opening remote text files for reading, ftputil will
  return byte strings from `read(line/s)` when run under
  Python 2 and unicode strings when run under Python 3.

    Pro: Each of the Python versions has ftputil behavior
    which matches the Python standard library behavior of
    the respective Python version.

    Con: Developers who want to use ftputil under Python 2
    _and_ 3 have to program against two different APIs since
    their code "inherits" ftputil's duality.

    Con: Support for two different APIs will make the
    ftputil code (at least a bit) more complicated than just
    returning unicode strings under both Python versions.

* Approach 2

  When opening remote text files for reading, ftputil will
  always return unicode strings from `read(line/s)`,
  regardless of whether it runs under Python 2 or Python 3.

    Pro: Uniform API, independent on underlying Python
    version.

    Pro: Supporting a single API will result in cleaner code
    in ftputil than when supporting different APIs (see
    above).

    Con: This approach might break some code which expects
    the returned strings under Python 2 to be byte strings.

    Con: Developers who only use Python 2 might be confused
    if ftputil returns unicode strings from `read(line/s)`
    since this behavior doesn't match files opened with
    `open` in Python 2.

Which approach do you recommend and why do you prefer that
approach? Are there other approaches I have overlooked? Do
you have other suggestions?

Best regards,
Stefan

[toc] | [next] | [standalone]

#43532

On 4/13/2013 12:36 PM, Stefan Schwarzer wrote:
> Hello,
>
> I'm currently changing the FTP client library ftputil [1]
> so that the same code of the library works with Python
> 2 (2.6 and up) and Python 3. (At the moment the code is for
> Python 2 only.) I've run into a API design issue where I
> don't know which API I should offer ftputil users under
> Python 2 and Python 3.
>
> [1] http://ftputil.sschwarzer.net/
>
> Some important background information: A key idea in ftputil
> is that it uses the same APIs as the Python standard library
> where possible. For example, with ftputil you can write code
> like this:
>
>    with ftputil.FTPHost(host, user, password) as ftp_host:
>        # Like `os.path.isdir`, but works on the FTP server.
>        if ftp_host.path.isdir("hello_dir"):
>            # Like `os.chdir`, but works on the FTP server.
>            ftp_host.chdir("hello_dir")
>            # Like the `open` builtin, but opens a remote file.
>            with ftp_host.open("new_file", "w") as fobj:
>                # Like `file.write` and `file.close`
>                fobj.write("Hello world!")
>                fobj.close()
>
> Since most of Python 2's and Python 3's filesystem-related
> APIs accept either bytes and character strings (and return
> the same type if they return anything string-like at all),
> the design here is rather clear to me.
>
> However, I have some difficulty with ftputil's counterpart
> of the `open` builtin function when files are opened for
> reading in text mode. Here are the approaches I've been
> thinking of so far:
>
> * Approach 1
>
>    When opening remote text files for reading, ftputil will
>    return byte strings from `read(line/s)` when run under
>    Python 2 and unicode strings when run under Python 3.
>
>      Pro: Each of the Python versions has ftputil behavior
>      which matches the Python standard library behavior of
>      the respective Python version.
>
>      Con: Developers who want to use ftputil under Python 2
>      _and_ 3 have to program against two different APIs since
>      their code "inherits" ftputil's duality.
>
>      Con: Support for two different APIs will make the
>      ftputil code (at least a bit) more complicated than just
>      returning unicode strings under both Python versions.
>
> * Approach 2
>
>    When opening remote text files for reading, ftputil will
>    always return unicode strings from `read(line/s)`,
>    regardless of whether it runs under Python 2 or Python 3.
>
>      Pro: Uniform API, independent on underlying Python
>      version.
>
>      Pro: Supporting a single API will result in cleaner code
>      in ftputil than when supporting different APIs (see
>      above).
>
>      Con: This approach might break some code which expects
>      the returned strings under Python 2 to be byte strings.
>
>      Con: Developers who only use Python 2 might be confused
>      if ftputil returns unicode strings from `read(line/s)`
>      since this behavior doesn't match files opened with
>      `open` in Python 2.
>
> Which approach do you recommend and why do you prefer that
> approach? Are there other approaches I have overlooked? Do
> you have other suggestions?

Approach 2 matches (or should match) io.open, which became builtin open 
in Python 3. I would simply document that ftp_host.open mimics io.open 
in the same way that ftp_host.chdir, etcetera, match os.chdir, etc. Your 
principle will remain intact.

Anyone writing *new* Py 2 code with any idea of ever running on Py 3 
should be using io.open anyway. That is why it was backported. You might 
be able to reuse some io code or subclass some io classes for your 
implementation.

[toc] | [prev] | [next] | [standalone]

#43535

On 04/13/2013 09:36 AM, Stefan Schwarzer wrote:
> * Approach 2
>
>    When opening remote text files for reading, ftputil will
>    always return unicode strings from `read(line/s)`,
>    regardless of whether it runs under Python 2 or Python 3.
>
>      Pro: Uniform API, independent on underlying Python
>      version.
>
>      Pro: Supporting a single API will result in cleaner code
>      in ftputil than when supporting different APIs (see
>      above).
>
>      Con: This approach might break some code which expects
>      the returned strings under Python 2 to be byte strings.
>
>      Con: Developers who only use Python 2 might be confused
>      if ftputil returns unicode strings from `read(line/s)`
>      since this behavior doesn't match files opened with
>      `open` in Python 2.
>
> Which approach do you recommend and why do you prefer that
> approach?

Approach 2, because it is much saner to deal with unicode inside the program, and only switch back to some kind of 
encoding when writing to files/pipes/etc.  Since you are going to support python 3 as well you can bump the major 
version number and note the backward incompatibility.

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

#43567

Terry, Ethan:

Thanks a lot for your excellent advice. :-)

On 2013-04-13 19:32, Terry Jan Reedy wrote:
> Approach 2 matches (or should match) io.open, which became
> builtin open in Python 3. I would simply document that
> ftp_host.open mimics io.open in the same way that
> ftp_host.chdir, etcetera, match os.chdir, etc. Your
> principle will remain intact.

I didn't know about `io.open` (or had forgotten it).

> Anyone writing *new* Py 2 code with any idea of ever
> running on Py 3 should be using io.open anyway. That is
> why it was backported. You might be able to reuse some io
> code or subclass some io classes for your implementation.

Since I use `socket.makefile` to create the underlying file
objects, I can use `BufferedReader`/`BufferedWriter` and
`TextIOWrapper` to supply buffering and encoding/decoding.

On 2013-04-13 20:03, Ethan Furman wrote:
> Approach 2, because it is much saner to deal with unicode
> inside the program, and only switch back to some kind of
> encoding when writing to files/pipes/etc.

Yes, this is a much saner design. I just was hesitant
because of the introduced backward incompatibility and
wanted to get other's opinions.

> Since you are going to support python 3 as well you can
> bump the major version number and note the backward
> incompatibility.

Actually I plan to increase the version number from 2.8 to
3.0 because of the Python 3 support and already intend to
change some module names that will be visible to client
code. So this is also a good opportunity to clean up the
file interface. :)

Best regards,
Stefan

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web