Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #107570

Re: Remove directory tree without following symlinks

Show key headers only | View raw

On Sun, Apr 24, 2016 at 5:42 AM, Albert-Jan Roskam
<sjeik_appie@hotmail.com> wrote:
> Aww, I kinda forgot about that already, but I came across this last
> year [1]. Apparently, shutil.rmtree(very_long_path) failed under Win 7,
> even with the "silly prefix". I believe very_long_path was a
> Python2-str.
> [1]
> https://mail.python.org/pipermail/python-list/2015-June/693156.html

Python 2's str branch of the os functions gets implemented on Windows
using the [A]NSI API, such as FindFirstFileA and FindNextFileA to
implement listdir(). Generally the ANSI API is a light wrapper around
the [W]ide-character API. It simply decodes byte strings to UTF-16 and
calls the wide-character function (or a common internal function).

IIRC, in Windows 7, byte strings are decoded using a per-thread buffer
with size MAX_PATH (260), so prefixing the path with "\\?\" won't
help. You have to use the wide-character API. Windows 10, on the other
hand, decodes using a dynamically allocated buffer, so you can usually
get away with using a long byte string. But not with Python 2
os.listdir(), which uses a stack-allocated MAX_PATH+5 buffer in the
str branch. For example:

Python 2 os.mkdir works:

    >>> path = os.path.normpath('//?/C:/Temp/long/' + 'a' * 255)
    >>> os.makedirs(path)

but os.listdir requires unicode:

    >>> os.listdir(path)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: must be (buffer overflow), not str
    >>> os.listdir(path.decode('mbcs'))
    []

Also, the str branch of listdir appends "/*.*", with a forward slash,
so it's incompatible with the "\\?\" prefix, even for short paths:

    >>> os.listdir(r'\\?\C:\Temp')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    WindowsError: [Error 123] The filename, directory name, or volume
    label syntax is incorrect: '\\\\?\\C:\\Temp/*.*'

> It seems useful if shutil or os.path would automatically prefix paths
> with "\\?\". It is rarely really needed, though. (in my case it was
> needed to copy a bunch of MS Outlook .msg files, which automatically
> get the subject line as the filename, and perhaps the first sentence
> of the mail of the mail has no subject).

I doubt a change like that would get backported to 2.7. Recently there
was a lengthy discussion about adding an __fspath__ protocol to Python
3. Possibly this can be automatically handled in the __fspath__
implementation of pathlib.WindowsPath and the DirEntry type returned
by os.scandir.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Remove directory tree without following symlinks Steven D'Aprano <steve@pearwood.info> - 2016-04-23 00:56 +1000
  Re: Remove directory tree without following symlinks Random832 <random832@fastmail.com> - 2016-04-22 11:09 -0400
    Re: Remove directory tree without following symlinks Steven D'Aprano <steve@pearwood.info> - 2016-04-23 03:14 +1000
      RE: Remove directory tree without following symlinks Albert-Jan Roskam <sjeik_appie@hotmail.com> - 2016-04-22 17:39 +0000
      Re: Remove directory tree without following symlinks eryk sun <eryksun@gmail.com> - 2016-04-22 13:28 -0500
      RE: Remove directory tree without following symlinks Albert-Jan Roskam <sjeik_appie@hotmail.com> - 2016-04-23 09:34 +0000
      Re: Remove directory tree without following symlinks eryk sun <eryksun@gmail.com> - 2016-04-23 15:22 -0500
      Re: Remove directory tree without following symlinks eryk sun <eryksun@gmail.com> - 2016-04-24 14:42 -0500
  Re: Remove directory tree without following symlinks Paul Rubin <no.email@nospam.invalid> - 2016-04-23 01:13 -0700
    Re: Remove directory tree without following symlinks Steven D'Aprano <steve@pearwood.info> - 2016-04-23 20:24 +1000
      Re: Remove directory tree without following symlinks Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2016-04-23 23:37 +1200
      Re: Remove directory tree without following symlinks Random832 <random832@fastmail.com> - 2016-04-23 17:04 -0400
  Re: Remove directory tree without following symlinks Nobody <nobody@nowhere.invalid> - 2016-04-23 17:29 +0100
    Re: Remove directory tree without following symlinks Random832 <random832@fastmail.com> - 2016-04-23 17:07 -0400

csiph-web