Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: eryk sun <eryksun@gmail.com>
Newsgroups: comp.lang.python
Subject: Re: Remove directory tree without following symlinks
Date: Sun, 24 Apr 2016 14:42:29 -0500
Lines: 57
Message-ID: <mailman.57.1461526992.32212.python-list@python.org>
References: <571a3ba2$0$1597$c3e8da3$5496439d@news.astraweb.com> <1461337766.365000.586700849.0DDBDB0B@webmail.messagingengine.com> <mailman.16.1461337768.2861.python-list@python.org> <571a5be6$0$1590$c3e8da3$5496439d@news.astraweb.com> <DUB123-W22A605538041FB2749C64B836F0@phx.gbl> <CACL+1aueH_scSTYW1Q6pGsZ92+8OWHB6tks3SqAAaU-wb6Ac4Q@mail.gmail.com> <DUB123-W362E7C64F347B92A74E34B83600@phx.gbl> <CACL+1avhu9YcM5qbFaAdhq9ZUoizP5Dc8p99LMrCcREmcxBNWw@mail.gmail.com> <DUB123-W25926A459EE112FA2741FA83610@phx.gbl> <CACL+1auZBa6q-+_=bbx-H6J7bpsXdYJ2dNx7mRpfeRqo_jru1Q@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
In-Reply-To: <DUB123-W25926A459EE112FA2741FA83610@phx.gbl>
Precedence: list
Xref: csiph.com comp.lang.python:107570

On Sun, Apr 24, 2016 at 5:42 AM, Albert-Jan Roskam
<sjeik_appie@hotmail.com> wrote:
> Aww, I kinda forgot about that already, but I came across this last
> year [1]. Apparently, shutil.rmtree(very_long_path) failed under Win 7,
> even with the "silly prefix". I believe very_long_path was a
> Python2-str.
> [1]
> https://mail.python.org/pipermail/python-list/2015-June/693156.html

Python 2's str branch of the os functions gets implemented on Windows
using the [A]NSI API, such as FindFirstFileA and FindNextFileA to
implement listdir(). Generally the ANSI API is a light wrapper around
the [W]ide-character API. It simply decodes byte strings to UTF-16 and
calls the wide-character function (or a common internal function).

IIRC, in Windows 7, byte strings are decoded using a per-thread buffer
with size MAX_PATH (260), so prefixing the path with "\\?\" won't
help. You have to use the wide-character API. Windows 10, on the other
hand, decodes using a dynamically allocated buffer, so you can usually
get away with using a long byte string. But not with Python 2
os.listdir(), which uses a stack-allocated MAX_PATH+5 buffer in the
str branch. For example:

Python 2 os.mkdir works:

    >>> path = os.path.normpath('//?/C:/Temp/long/' + 'a' * 255)
    >>> os.makedirs(path)

but os.listdir requires unicode:

    >>> os.listdir(path)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: must be (buffer overflow), not str
    >>> os.listdir(path.decode('mbcs'))
    []

Also, the str branch of listdir appends "/*.*", with a forward slash,
so it's incompatible with the "\\?\" prefix, even for short paths:

    >>> os.listdir(r'\\?\C:\Temp')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    WindowsError: [Error 123] The filename, directory name, or volume
    label syntax is incorrect: '\\\\?\\C:\\Temp/*.*'

> It seems useful if shutil or os.path would automatically prefix paths
> with "\\?\". It is rarely really needed, though. (in my case it was
> needed to copy a bunch of MS Outlook .msg files, which automatically
> get the subject line as the filename, and perhaps the first sentence
> of the mail of the mail has no subject).

I doubt a change like that would get backported to 2.7. Recently there
was a lengthy discussion about adding an __fspath__ protocol to Python
3. Possibly this can be automatically handled in the __fspath__
implementation of pathlib.WindowsPath and the DirEntry type returned
by os.scandir.