Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=BoWTiC4idk0U1STPVvCWf2AwSNm5Oc4/2Uh+/K05RW0VKKUrwoQ1shbUEKZisIfsc2UXBNj3gJQafb5RMOjlWEn2VebjGaeelRieDlTb0Lvc/HZzv035lhBcxwdfZtl3TaG7W/ezQkfK86FyWq286b3ZOa/i9UOuaCSmOgPuRuY=;
References: <5286054F.6000707@chamonix.reportlab.co.uk> <1384539385.17784.47988289.02AE655C@webmail.messagingengine.com> <5289FE6C.7030007@chamonix.reportlab.co.uk> <528A092D.4060408@chamonix.reportlab.co.uk> <1384813806.25855.49113337.110F19E2@webmail.messagingengine.com>
Date: Mon, 18 Nov 2013 21:27:41 -0800 (PST)
From: Andrew Barnert <abarnert@yahoo.com>
Subject: Re: [Python-ideas] Unicode stdin/stdout
To: "random832@fastmail.us" <random832@fastmail.us>, Robin Becker <robin@reportlab.com>, "python-ideas@python.org" <python-ideas@python.org>
In-Reply-To: <1384813806.25855.49113337.110F19E2@webmail.messagingengine.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Cc: "python-list@python.org" <python-list@python.org>
Precedence: list
Reply-To: Andrew Barnert <abarnert@yahoo.com>
Newsgroups: comp.lang.python
Message-ID: <mailman.2882.1384839071.18130.python-list@python.org>
Lines: 60
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:59944

From: "random832@fastmail.us" <random832@fastmail.us>=0A=0A=0A=0A> On Mon, =
Nov 18, 2013, at 7:33, Robin Becker wrote:=0A>>  UTF-8 stuff=0A> =0A> This =
doesn't really solve the issue I was referring to, which is that=0A> window=
s _console_ (i.e. not redirected file or pipe) I/O can only=0A> support uni=
code via wide character (UTF-16) I/O with a special function,=0A> not via u=
sing byte-based I/O with the normal write function.=0A=0A=0AThe problem is =
that Windows 16-bit I/O doesn't fit into the usual io module hierarchy. Not=
 because it uses an encoding of UTF-16 (although anyone familiar with ReadC=
onsoleW/WriteConsoleW from other languages may be a bit confused that Pytho=
n's lowest-level wrappers around them deal in byte counts instead of WCHAR =
counts), but because you have to use HANDLEs instead of fds. So, there are =
going to be some compromises and some complexity.=0A=0AOne possibility is t=
o use as much of the io hierarchy as possible, but not try to make it flexi=
ble enough to be reusable for arbitrary HANDLEs: Add=C2=A0WindowsFileIO and=
 WindowsConsoleIO classes that implement RawIOBase with a native HANDLE and=
 ReadFile/WriteFile and ReadConsoleW/WriteConsoleW respectively. Both work =
in terms of bytes (which means WindowsConsoleIO.read has to //2 its argumen=
t, and write has to *2 the result). You also need a create_windows_io funct=
ion that wraps a HANDLE by calling GetConsoleMode and constructing a Window=
sConsoleIO or WindowsFileIO as appropriate, then creates a BufferedReader/W=
riter around that, then constructs a TextIOWrapper with UTF-16 or the defau=
lt encoding around that. At startup, you just do that for the three GetStdH=
andle handles, and that's your stdin, stdout, and stderr.=0A=0ABesides not =
being reusable enough for people who want to wrap HANDLEs from other librar=
ies or attach to new consoles from Python, it's not clear what fileno() sho=
uld return. You could fake it and return the MSVCRT fds that correspond to =
the same files as the HANDLEs, but it's possible to end up with one redirec=
ted and not the other (e.g., if you detach the console), and I'm not sure w=
hat happens if you mix and match the two. A more "correct" solution would b=
e to call _open_osfhandle on=C2=A0the HANDLE (and then keep track of the fa=
ct that os.close closes the HANDLE, or leave it up to the user to deal with=
 bad handle errors?), but I'm not sure that's any better in practice. Also,=
 should a console HANDLE use _O_WTEXT for its fd (in which case the user ha=
s to know that he has a _O_WTEXT handle even though there's no way to see t=
hat from Python), or not (in which case he's mixing 8-bit and 16-bit I/O on=
 the same file)?=0A=0AIt might be reasonable to just not expose fileno(); m=
ost code that wants the fileno() for stdin is just going to do something Un=
ix-y that's not going to work anyway (select it, tcsetattr it, pass it over=
 a socket to another file, =E2=80=A6).=0A=0AA different approach would be t=
o reuse as _little_ of io as possible, instead of as much: Windows stdin/st=
dout/stderr could each be custom TextIOBase implementations that work strai=
ght on HANDLEs and don't=C2=A0even support buffer (or detach), much less fi=
leno. That exposes even less functionality to users, of course. It also mea=
ns we need a parallel implementation of all the buffering logic. (On the ot=
her hand, it also leaves the door open to expose some Windows functionality=
, like async ReadFileEx/WriteFileEx, in a way that would be very hard throu=
gh the normal layers=E2=80=A6)=0A=0A=0AIt shouldn't be too hard to write mo=
st of these via an extension module or ctypes to experiment with it. As lon=
g as you're careful not to mix winsys.stdout and sys.stdout (the module cou=
ld even set sys.stdin, sys.stdout, sys.stderr=3Dstdin, stdout, stderr at im=
port time, or just del them, for a bit of protection), it should work.=0A=
=0AIt might be worth implementing a few different designs to play with, and=
 putting them through their paces with some modules and scripts that do dif=
ferent things with stdio (including running the scripts with cmd.exe redire=
cted I/O and with subprocess PIPEs) to see which ones have problems or limi=
tations that are hard to foresee in advance.=0A=0AIf you have a design that=
 you think sounds good, and are willing to experiment the hell out of it, a=
nd don't know how to get started but would be willing to debug and finish a=
 mostly-written/almost-working implementation, I could slap something toget=
her with ctypes to get you started.