Re: Filesystem syntax constraints under Windows

From	David Brown <david.brown@hesbynett.no>
Newsgroups	comp.arch.embedded
Subject	Re: Filesystem syntax constraints under Windows
Date	2014-10-22 13:11 +0200
Organization	A noiseless patient Spider
Message-ID	<m283df$s7t$1@dont-email.me> (permalink)
References	(3 earlier) <m26m8c$ai5$1@dont-email.me> <m26o1u$2jg$1@speranza.aioe.org> <m26qd9$min$1@dont-email.me> <m27hov$7ks$1@dont-email.me> <m27rng$4am$1@dont-email.me>

Show all headers | View raw

On 22/10/14 11:00, Dimiter_Popoff wrote:
> On 22.10.2014 г. 09:10, David Brown wrote:
>> On 22/10/14 01:32, Dimiter_Popoff wrote:
>>> Hi Don,
>>>
>>> On 22.10.2014 г. 01:51, Don Y wrote:
>>>> ...
>>>>
>>>> Under Interix, you bypass Windows' rules for names and write
>>>> directly to
>>>> the file system (disk media).
>>>
>>> Well writing to a directory entry not through a system call would
>>> easily break the directory, sure. I don't have to tell you this
>>> is not the way you want to go in an end product (instability,
>>> impredictability issues - how will the next OS version treat
>>> these invalid entries etc. etc.).
>>>
>>
>> He is not hacking the disk in some way - he is using Interix, which is
>> written by Microsoft as sort of mostly Posix compatibility layer for
>> Windows.  This layer does not use the Win32 API, but it uses the NT
>> kernel services and system calls in the same way that the Win32 API
>> does.  Thus is uses posix-compatible API's to ask the Windows VFS system
>> to create, read or write files, and the VFS system passes this on to the
>> NTFS system.  When using explorer, the code uses thw Win32 API to talk
>> to the VFS and then on to the NTFS.
>>
>> What we are seeing here is that the NTFS filesystem is perfectly capable
>> of holding filenames with almost arbitrary characters, and does not do
>> any case-dependent handling (thus "a" and "A" are different characters,
>> and can be different filenames).  This is not surprising, since NTFS was
>> designed to be usable in a Posix environment, and also since it uses a
>> restricted UTF-16 (no multi-point characters) for filenames and does not
>> attempt to include the vast set of rules needed to handle case
>> dependencies.
>>
>> We also see that through Interix, filenames are not mangled, except
>> perhaps to handle "/" as a directory separator.  Through the Win32 API,
>> filenames are mangled in a variety of ways both going into the VFS, and
>> coming out of it - and can be mangled in different ways depending on the
>> particular calls being used.  They are then further mangled by the
>> application ("explorer.exe", "cmd.exe", etc.).  This is no surprise
>> either, given the history of the system which attempts to remain
>> somewhat compatible with a range of different limitations in kernels and
>> filesystems DOS, Win9x, NT, FAT, FAT32, etc.
>>
>> And since the mangling and translations are done at different stages -
>> some in the APIs, some in the applications, some in the libraries -
>> there is repetition and inconsistencies.  This is also no surprise,
>> based on the development environment at MS - different groups handle
>> different parts, but act competitively rather than cooperatively, with
>> an appalling lack of documentation or references.
>>
> 
> David, I am not sure what you are trying to explain but I do not think
> there are many people here who need to be told that a set of characters
> stored as bytes can be compared in a case dependent or independent
> manner.

I am trying to explain that Don is not doing something odd or outside of
the windows system here, as you seemed to think:

>>> Well writing to a directory entry not through a system call would
>>> easily break the directory, sure.

If he had used a disk editor to directly change the filenames, then I
could understand your comment.  But he has not done anything like that -
he has used programs written by Microsoft to run on Windows, and used
them to create filenames that other parts of Windows can't deal with
properly.


> Key to the point is:
> 1. The filesystem stores all name related data it is given (i.e. without
> loss of information),

Yes...

> 2. The user is not exposed to the bitstreams the OS stores but to to
> text which consists of characters which are part of an alphabet,
> for example the Latin alphabet as used in English has 26 characters.

In other words, Windows mangles the names it is given.

> 
> Don's problem is that he just cannot copy a unix directory if it
> contains duplicate file names to an NTFS one such that it is usable.

That is one of his problems, yes.

> Of course he can hack his way into doing it, whether through some
> MS written hack which you say is not a hack or otherwise.

The Win32 API allows files to be created or opened using "posix
semantics" for filenames, including case-sensitive files, characters
such as ":" and "*" in filenames, and multiple files differing only in
the case of their names.  Even if you want to call the MS-supplied posix
compatibility layer a "hack", I don't think the standard Win32 API is a
hack.

> 
> The problem remains and will remain, as unix does
> not output names but file identifiers (names consist of text, remember
> the alphabet and the character count). The fact that these identifiers
> have been misused as text for decades does not mean much beyond
> the expectations of hardcore unix users that the English alphabet
> will suddenly begin to have 52 characters.

This goes back to your unique idea that files have a sort of colloquial
human-friendly nick-name that is a different concept from their
"filename" that everyone else uses.

If we were to accept that idea, then /all/ systems have that "problem" -
because no system will be happy with a file system that uses approximate
names instead of concrete identifiers.  By that I mean that
"index.html", "Index.html", "Index.html", "Index", "The index file", and
"The first page" are perfectly good human-friendly names for the first
page of a website - but no OS or filesystem would accept them as
alternatives for a file identified as "index.html".

> 
> What exactly are you trying to prove, why do you keep on flailing.
> Why is it so hard for you to accept that you have overlooked a few
> simple, obvious facts and just move on.
> 

I am just trying to correct your (apparent) misunderstanding about what
Don was doing, and how Windows and NTFS treat filenames.

Thread

Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-10 00:31 -0700
  Re: Filesystem syntax constraints under Windows glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2014-10-10 09:06 +0000
    Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-10 13:32 +0200
      Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-10 07:23 -0700
        Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-10 07:26 -0700
          Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-10 19:17 +0300
            Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-10 19:12 +0200
              Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-10 20:34 +0300
              Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-10 11:32 -0700
            Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-10 10:53 -0700
              Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-10 21:24 +0300
                Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-10 13:36 -0700
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-11 00:18 +0300
                Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-10 15:28 -0700
                Re: Filesystem syntax constraints under Windows Tonton Th <tTh@nowhere.invalid> - 2014-10-12 20:07 +0000
                Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-12 19:52 -0700
              Re: Filesystem syntax constraints under Windows Hans-Bernhard Bröker <HBBroeker@t-online.de> - 2014-10-10 21:15 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-10 23:04 +0300
            Re: Filesystem syntax constraints under Windows Stefan Reuther <stefan.news@arcor.de> - 2014-10-11 11:28 +0200
              Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-11 17:13 +0300
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-11 17:26 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-11 18:51 +0300
                Re: Filesystem syntax constraints under Windows Stefan Reuther <stefan.news@arcor.de> - 2014-10-12 12:04 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-12 15:59 +0300
                Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-12 08:04 -0700
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-12 18:47 +0300
                Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-12 09:01 -0700
                Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-12 09:05 -0700
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-12 19:17 +0300
                Re: Filesystem syntax constraints under Windows Stefan Reuther <stefan.news@arcor.de> - 2014-10-13 19:26 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-13 22:00 +0300
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-14 10:25 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-14 13:28 +0300
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-14 14:37 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-14 16:00 +0300
                Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-14 06:19 -0700
                Re: Filesystem syntax constraints under Windows Hans-Bernhard Bröker <HBBroeker@t-online.de> - 2014-10-15 19:57 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-16 00:06 +0300
                Re: Filesystem syntax constraints under Windows Stefan Reuther <stefan.news@arcor.de> - 2014-10-16 18:16 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-16 20:49 +0300
                Re: Filesystem syntax constraints under Windows Stefan Reuther <stefan.news@arcor.de> - 2014-10-14 18:59 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-14 21:39 +0300
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-14 23:29 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-15 00:46 +0300
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-15 01:38 +0200
        Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-10 17:31 +0200
          Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-10 10:34 -0700
    Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-10 06:59 -0700
  Re: Filesystem syntax constraints under Windows Stefan Reuther <stefan.news@arcor.de> - 2014-10-10 18:42 +0200
    Re: Filesystem syntax constraints under Windows glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2014-10-10 19:00 +0000
    Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-10 13:43 -0700
      Re: Filesystem syntax constraints under Windows upsidedown@downunder.com - 2014-10-11 11:06 +0300
        Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-11 01:35 -0700
        Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-11 17:24 +0300
        Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-11 17:43 +0200
          Re: Filesystem syntax constraints under Windows Stefan Reuther <stefan.news@arcor.de> - 2014-10-12 12:07 +0200
            Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-12 23:41 +0200
              Re: Filesystem syntax constraints under Windows Stefan Reuther <stefan.news@arcor.de> - 2014-10-13 19:14 +0200
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-13 20:18 +0200
                Re: Filesystem syntax constraints under Windows Mel Wilson <mwilson@the-wire.com> - 2014-10-15 18:43 +0000
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-15 21:58 +0200
                Re: Filesystem syntax constraints under Windows George Neuner <gneuner2@comcast.net> - 2014-10-15 23:55 -0400
    Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-20 09:45 -0700
      Re: Filesystem syntax constraints under Windows Stefan Reuther <stefan.news@arcor.de> - 2014-10-21 18:47 +0200
        Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-21 15:38 -0700
          Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-21 17:51 -0700
          Re: Filesystem syntax constraints under Windows George Neuner <gneuner2@comcast.net> - 2014-10-22 01:35 -0400
            Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-21 23:22 -0700
              Re: Filesystem syntax constraints under Windows Stefan Reuther <stefan.news@arcor.de> - 2014-10-22 18:39 +0200
                Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-22 11:34 -0700
                Re: Filesystem syntax constraints under Windows upsidedown@downunder.com - 2014-10-22 22:11 +0300
            Re: Filesystem syntax constraints under Windows glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2014-10-23 00:20 +0000
              Re: Filesystem syntax constraints under Windows George Neuner <gneuner2@comcast.net> - 2014-10-23 01:36 -0400
                Re: Filesystem syntax constraints under Windows glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2014-10-23 09:08 +0000
                Re: Filesystem syntax constraints under Windows George Neuner <gneuner2@comcast.net> - 2014-10-23 14:50 -0400
                Re: Filesystem syntax constraints under Windows Hans-Bernhard Bröker <HBBroeker@t-online.de> - 2014-10-23 21:42 +0200
      Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-22 01:21 +0300
        Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-21 15:51 -0700
          Re: Filesystem syntax constraints under Windows glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2014-10-21 23:27 +0000
            Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-21 16:50 -0700
            Re: Filesystem syntax constraints under Windows Hans-Bernhard Bröker <HBBroeker@t-online.de> - 2014-10-22 21:00 +0200
              Re: Filesystem syntax constraints under Windows upsidedown@downunder.com - 2014-10-22 22:54 +0300
          Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-22 02:32 +0300
            Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-21 17:21 -0700
              Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-22 03:52 +0300
                Re: Filesystem syntax constraints under Windows Don Y <this@is.not.me.com> - 2014-10-21 21:13 -0700
            Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-22 08:10 +0200
              Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-22 12:00 +0300
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-22 13:11 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-22 15:19 +0300
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-22 15:21 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-22 17:26 +0300
                Re: Filesystem syntax constraints under Windows Stefan Reuther <stefan.news@arcor.de> - 2014-10-22 18:46 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-23 02:37 +0300
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-22 23:09 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-23 03:06 +0300
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-23 10:18 +0200
                Re: Filesystem syntax constraints under Windows Dimiter_Popoff <dp@tgi-sci.com> - 2014-10-23 13:56 +0300
                Re: Filesystem syntax constraints under Windows David Brown <david.brown@hesbynett.no> - 2014-10-23 15:59 +0200

csiph-web