Re: How to waste computer memory?

From	Marko Rauhamaa <marko@pacujo.net>
Newsgroups	comp.lang.python
Subject	Re: How to waste computer memory?
Date	2016-03-19 11:31 +0200
Organization	A noiseless patient Spider
Message-ID	<87lh5en79a.fsf@elektro.pacujo.net> (permalink)
References	(5 earlier) <87twk3oli0.fsf@elektro.pacujo.net> <mailman.351.1458332168.12893.python-list@python.org> <87k2kzo5y5.fsf@elektro.pacujo.net> <mailman.353.1458335305.12893.python-list@python.org> <56ed0a71$0$1607$c3e8da3$5496439d@news.astraweb.com>

Show all headers | View raw

Steven D'Aprano <steve@pearwood.info>:

> One thing that NTFS gets right is that all path names are guaranteed
> to be well-formed, valid Unicode. I believe that they are stored in
> UTF-16, and unlike the ext file systems used on Linux, they are not
> arbitrary bytes.

<URL: https://msdn.microsoft.com/en-us/library/windows/desktop/dd31774
8%28v=vs.85%29.aspx> states that NTFS filenames disallow '\', '/', '.',
'?', '*' as well as '¥'. Apparently the ban on the yen symbol isn't
enforced by the FS.

I haven't found a direct statement whether NTFS internally enforces the
soundness of UTF-16 or if it is simply doing UCS-2.

<URL: https://msdn.microsoft.com/en-us/library/windows/desktop/dd374069
%28v=vs.85%29.aspx>:

   Using the surrogate mechanism, UTF-16 can support all 1,114,112
   potential Unicode characters.

But Unicode doesn't contain 1,114,112 characters—the surrogates are
excluded from Unicode, and definitely cannot be encoded using
UTF-anything.

Furthermore, the page notes:

   Note Windows 2000 introduces support for basic input, output, and
   simple sorting of supplementary characters. However, not all system
   components are compatible with supplementary characters.

(Somewhat related, Python doesn't enforce the soundness of Unicode
because Python allows surrogate code points in strings.)

> I believe that HFS+ on Apple Macs goes one step further and guarantees
> that paths are always fully normalised, so that it's impossible to
> have (e.g.) two files ã (U+00E3 LATIN SMALL LETTER A WITH TILDE) and ã
> (U+0061 LATIN SMALL LETTER A + U+0303 COMBINING TILDE) in the same
> directory.
>
> Unfortunately, backwards compatibility is holding Linux file systems
> back...

Linux got lucky by not jumping the gun. We are still waiting for the
dust to settle.

Unicode made several (understandable but grave) mistakes along the way:

   * UCS-2

   * supplementary code points

   * BOM

   * endianness

   * normalization

We still don't know if the final result will be UCS-4 everywhere (with
all 2**32 code points allowed?!) or UTF-8 everywhere.


Marko

Thread

How to waste computer memory? wxjmfauth@gmail.com - 2016-03-17 07:34 -0700
  Re: How to waste computer memory? Rick Johnson <rantingrickjohnson@gmail.com> - 2016-03-17 12:21 -0700
    Re: How to waste computer memory? cl@isbd.net - 2016-03-17 20:31 +0000
      Re: How to waste computer memory? Chris Angelico <rosuav@gmail.com> - 2016-03-18 07:42 +1100
        Re: How to waste computer memory? Grant Edwards <invalid@invalid.invalid> - 2016-03-17 21:08 +0000
          Re: How to waste computer memory? Chris Angelico <rosuav@gmail.com> - 2016-03-18 08:13 +1100
            Re: How to waste computer memory? Paul Rubin <no.email@nospam.invalid> - 2016-03-17 14:30 -0700
          Re: How to waste computer memory? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2016-03-17 22:32 +0000
          Re: How to waste computer memory? cl@isbd.net - 2016-03-17 22:42 +0000
        Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-17 23:11 +0200
          Re: How to waste computer memory? Chris Angelico <rosuav@gmail.com> - 2016-03-18 08:17 +1100
          Re: How to waste computer memory? BartC <bc@freeuk.com> - 2016-03-17 21:26 +0000
            Re: How to waste computer memory? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2016-03-17 22:38 +0000
            Re: How to waste computer memory? Chris Angelico <rosuav@gmail.com> - 2016-03-18 10:02 +1100
        Re: How to waste computer memory? alister <alister.ware@ntlworld.com> - 2016-03-17 21:37 +0000
          Re: How to waste computer memory? alister <alister.ware@ntlworld.com> - 2016-03-17 21:43 +0000
          Re: How to waste computer memory? Gene Heskett <gheskett@wdtv.com> - 2016-03-17 20:51 -0400
            Re: How to waste computer memory? Rick Johnson <rantingrickjohnson@gmail.com> - 2016-03-17 18:47 -0700
            Re: How to waste computer memory? cl@isbd.net - 2016-03-18 10:44 +0000
              Re: How to waste computer memory? Gene Heskett <gheskett@wdtv.com> - 2016-03-18 10:11 -0400
              Re: How to waste computer memory? Grant Edwards <invalid@invalid.invalid> - 2016-03-19 13:50 +0000
    Re: How to waste computer memory? Ian Kelly <ian.g.kelly@gmail.com> - 2016-03-18 01:00 -0600
      Re: How to waste computer memory? Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-03-18 10:26 +0200
        Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-18 17:26 +0200
          Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-19 03:58 +1100
          Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-18 23:02 +0200
            Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-18 23:28 +0200
              Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-19 00:03 +0200
                Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-19 09:49 +0200
                Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-19 10:22 +0200
                Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-19 11:40 +0200
                Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-19 19:38 +1100
            Re: How to waste computer memory? wxjmfauth@gmail.com - 2016-03-19 00:14 -0700
              Re: How to waste computer memory? wxjmfauth@gmail.com - 2016-03-19 02:17 -0700
            Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-19 19:14 +1100
              Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-19 11:31 +0200
                Re: How to waste computer memory? wxjmfauth@gmail.com - 2016-03-19 03:40 -0700
                Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-19 13:07 +0200
                Re: How to waste computer memory? BartC <bc@freeuk.com> - 2016-03-19 12:24 +0000
                Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-19 14:43 +0200
                Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-20 01:18 +1100
                Re: How to waste computer memory? BartC <bc@freeuk.com> - 2016-03-19 15:14 +0000
                Re: How to waste computer memory? BartC <bc@freeuk.com> - 2016-03-19 15:20 +0000
                Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-19 22:32 +1100
                Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-19 14:42 +0200
                Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-20 01:39 +1100
                Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-19 16:56 +0200
                Re: How to waste computer memory? wxjmfauth@gmail.com - 2016-03-19 07:01 -0700
                Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-20 01:56 +1100
                Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-19 17:02 +0200
                Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-20 02:47 +1100
                Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-19 18:12 +0200
                Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-20 16:01 +1100
                Re: How to waste computer memory? Rustom Mody <rustompmody@gmail.com> - 2016-03-19 23:20 -0700
                Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-20 22:06 +1100
                Re: How to waste computer memory? Chris Angelico <rosuav@gmail.com> - 2016-03-20 22:22 +1100
                Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-20 23:14 +1100
                Re: How to waste computer memory? Chris Angelico <rosuav@gmail.com> - 2016-03-20 23:27 +1100
                Re: How to waste computer memory? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2016-03-20 14:55 +0000
                Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-20 17:36 +0200
                Re: How to waste computer memory? Random832 <random832@fastmail.com> - 2016-03-20 14:17 -0400
                Re: How to waste computer memory? Marko Rauhamaa <marko@pacujo.net> - 2016-03-20 09:30 +0200
      Re: How to waste computer memory? wxjmfauth@gmail.com - 2016-03-18 03:50 -0700
      Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-18 22:46 +1100
        Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-18 22:58 +1100
          Re: How to waste computer memory? wxjmfauth@gmail.com - 2016-03-18 12:53 -0700
        Re: How to waste computer memory? Chris Angelico <rosuav@gmail.com> - 2016-03-18 23:37 +1100
        Re: How to waste computer memory? Ian Kelly <ian.g.kelly@gmail.com> - 2016-03-18 07:57 -0600
    Re: How to waste computer memory? Steven D'Aprano <steve@pearwood.info> - 2016-03-19 03:44 +1100
      Re: How to waste computer memory? Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-03-18 20:22 +0200
        Re: How to waste computer memory? wxjmfauth@gmail.com - 2016-03-18 13:03 -0700
  Re: How to waste computer memory? sohcahtoa82@gmail.com - 2016-03-18 11:18 -0700

csiph-web