Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.sys.mac.system > #62250 > unrolled thread

Why Mavericks is slow or fast or needs an SSD

Back to article view | Back to comp.sys.mac.system

Contents

  Why Mavericks is slow or fast or needs an SSD Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2014-07-14 21:38 -0700
    Re: Why Mavericks is slow or fast or needs an SSD Jolly Roger <jollyroger@pobox.com> - 2014-07-15 16:02 +0000
      Re: Why Mavericks is slow or fast or needs an SSD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2014-07-16 04:16 +0000
        Re: Why Mavericks is slow or fast or needs an SSD Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2014-07-15 22:17 -0700
          Re: Why Mavericks is slow or fast or needs an SSD Jolly Roger <jollyroger@pobox.com> - 2014-07-16 05:45 +0000
          Re: Why Mavericks is slow or fast or needs an SSD Tim McNamara <timmcn@bitstream.net> - 2014-07-17 17:40 -0500
    Re: Why Mavericks is slow or fast or needs an SSD Backdrop - 2014-07-16 04:16 +0200
      Re: Why Mavericks is slow or fast or needs an SSD Jolly Roger <jollyroger@pobox.com> - 2014-07-16 05:51 +0000
    Re: Why Mavericks is slow or fast or needs an SSD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2014-07-15 23:11 -0400
      Re: Why Mavericks is slow or fast or needs an SSD nospam <nospam@nospam.invalid> - 2014-07-15 23:59 -0400
      Re: Why Mavericks is slow or fast or needs an SSD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2014-07-16 04:31 +0000
        Re: Why Mavericks is slow or fast or needs an SSD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2014-07-16 01:06 -0400
          Re: Why Mavericks is slow or fast or needs an SSD dempson@actrix.gen.nz (David Empson) - 2014-07-16 19:39 +1200
            Re: Why Mavericks is slow or fast or needs an SSD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2014-07-16 15:45 -0400
              Re: Why Mavericks is slow or fast or needs an SSD dempson@actrix.gen.nz (David Empson) - 2014-07-18 13:06 +1200
                Re: Why Mavericks is slow or fast or needs an SSD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2014-07-17 21:35 -0400
                  Re: Why Mavericks is slow or fast or needs an SSD dempson@actrix.gen.nz (David Empson) - 2014-07-18 18:03 +1200
          Re: Why Mavericks is slow or fast or needs an SSD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2014-07-16 16:37 +0000
    Re: Why Mavericks is slow or fast or needs an SSD Alan Browne <alan.browne@FreelunchVideotron.ca> - 2014-07-16 08:08 -0400

#62250 — Why Mavericks is slow or fast or needs an SSD

The Internet net is full of people who claim that Mavericks is much 
faster or unusably slow.  The truth is - it's both.  Here's some history 
on what's going on and why you should think twice about investing in 
more Apple hardware.


10.6:  This is often cited as the most elegant and most efficient OS X 
to date.  It's generally very fast and it handles heavy duty tasks well.  
There's the occasional case where it does badly - runaway RAM hogs.  A 
process demanding a very large amount of RAM was able to steal almost 
all of it away from everything else.  Idle processes would entirely 
"swap out" to disk and deactivate.  Windows buffers would even page out 
so that dragging a window caused it to shimmer as it paged back in.  All 
of this was good if a process needed the RAM momentarily to finish a 
task but it also meant that a runaway RAM hog required a power cycle.


10.7/10.8 : A solution is created for the runaway RAM hog: As of 10.7, 
no process is allowed to consume more than some fraction of total RAM 
and idle processes are no longer swapped out.  This backfires horribly.  
Processes that need large amounts of temporary RAM are forced to swap 
within their tiny allotment.  All of this paging monopolizes the 
filesystem so that the new features needing continuous filesystem 
access, like Auto Save, Auto Restore, Versions, and Mobile Backup, bring 
all Apple applications to a halt.  Old HFS+ isn't able to handle this.  
A process that needs temporary RAM takes 10x to 100x longer to finish 
than it did under 10.6, and new Apple apps are crippled the whole time.


10.9 : The broken VM tuning from 10.7 is fixed and augmented with 
compression.  VM compression consumes some RAM of its own but it 
generally outperforms 10.6 virtual memory.  Apple also adds a fix for 
processes that monopolize the filesystem: Each process is throttled.  
You can see this in 'fs_usage' as log entry "THROTTLED."  Again, a new 
fix backfires horribly.  Processes that shouldn't be throttled and don't 
need to be throttled still get throttled.  In my case, applications were 
getting just 300KB/sec of filesystem throughput; a 97% loss compared to 
10.8 and worse again compared to 10.6.  People who install an SSD notice 
a vast improvement while people who already have an SSD notice that it 
still seems slow.  What's going on?  Check these system parameters:


debug.lowpri_throttle_max_iosize: 131072
debug.lowpri_throttle_tier1_window_msecs: 25
debug.lowpri_throttle_tier2_window_msecs: 100
debug.lowpri_throttle_tier3_window_msecs: 500
debug.lowpri_throttle_tier1_io_period_msecs: 15
debug.lowpri_throttle_tier2_io_period_msecs: 50
debug.lowpri_throttle_tier3_io_period_msecs: 200
debug.lowpri_throttle_tier1_io_period_ssd_msecs: 5
debug.lowpri_throttle_tier2_io_period_ssd_msecs: 15
debug.lowpri_throttle_tier3_io_period_ssd_msecs: 25
debug.lowpri_throttle_enabled: 1



These parameters idle processes to limit their filesystem use.  Apple 
has provided a less aggressive throttle if an SSD is being used, which 
is why an SSD helps in cases where spinning rust should suffice.

Fire up your favorite disk intensive task and wait for the throughput to 
fall to nearly nothing.  Now run:

sudo sysctl -w debug.lowpri_throttle_enabled=0

Boom! Throughput is back!  Now all the Apple apps are beachballing 
again, but at least the computer isn't sitting idle any more.


All of this throttling is a very poor workaround to what has been needed 
for many years: A new filesystem.  HFS+ is too old, too slow, has too 
many caching bugs, too many journaling bugs, and produces complex 
filesystems that corrupt easily.  As of Yosemite, there's still the same 
old filesystem in place crippling the OS.  Your new $6000 Mac Pro is 
going to sit idle every time heavy disk I/O is demanded.

[toc] | [next] | [standalone]

#62260

On 2014-07-15, Kevin McMurtrie <mcmurtrie@pixelmemory.us> wrote:
> The Internet net is full of people who claim that Mavericks is much 
> faster or unusably slow.  The truth is - it's both.  Here's some history 
> on what's going on and why you should think twice about investing in 
> more Apple hardware.

[claims of sluggish performance and stalled apps, along with details
about VM management]

Meanwhile, I'm seeing no such problems with 10.6, 10.7, 10.8, or 10.9.
Go figure.

-- 
E-mail sent to this address may be devoured by my ravenous SPAM filter.
I often ignore posts from Google. Use a real news client instead.

JR

[toc] | [prev] | [next] | [standalone]

#62267

Okay, so one time? In band camp? Jolly Roger <jollyroger@pobox.com> was all, like: 
 --> 	 15 Jul 2014 16:02:52 GMT <c2l1lbFoajbU1@mid.individual.net>
> On 2014-07-15, Kevin McMurtrie <mcmurtrie@pixelmemory.us> wrote:
>> The Internet net is full of people who claim that Mavericks is much 
>> faster or unusably slow.  The truth is - it's both.  Here's some history 
>> on what's going on and why you should think twice about investing in 
>> more Apple hardware.

> [claims of sluggish performance and stalled apps, along with details
> about VM management]

> Meanwhile, I'm seeing no such problems with 10.6, 10.7, 10.8, or 10.9.
> Go figure.

McNuttie is a one-note bore.

But he is right about HFS+, it's long in the tooth and we need something
better.

OTOH, Swift came right out of the blue, so many some next-gen working
fileOS like zfs but not zfs is on the horizon.

-- 
Hey, baby, I've got just the cure for that penis envy back at my
apartment...

[toc] | [prev] | [next] | [standalone]

#62272

In article <slrnlsbv5b.1die.g.kreme@Amelia.local>,
 Lewis <g.kreme@gmail.com.dontsendmecopies> wrote:

> Okay, so one time? In band camp? Jolly Roger <jollyroger@pobox.com> was all, 
> like: 
>  --> 	 15 Jul 2014 16:02:52 GMT <c2l1lbFoajbU1@mid.individual.net>
> > On 2014-07-15, Kevin McMurtrie <mcmurtrie@pixelmemory.us> wrote:
> >> The Internet net is full of people who claim that Mavericks is much 
> >> faster or unusably slow.  The truth is - it's both.  Here's some history 
> >> on what's going on and why you should think twice about investing in 
> >> more Apple hardware.
> 
> > [claims of sluggish performance and stalled apps, along with details
> > about VM management]
> 
> > Meanwhile, I'm seeing no such problems with 10.6, 10.7, 10.8, or 10.9.
> > Go figure.
> 
> McNuttie is a one-note bore.

Apple is the one one-note bore.  They've been reliving the glory of 
their first iPhone for 7 years now, like a fat middle-aged person still 
wearing his lucky high school football jacket.  Their desktop systems 
are dying and iPhones are underwhelming.

I stopped watching Apple's big announcements years ago.  I don't care 
about iCloud and it's thimble of storage that isn't compatible with 
anything.  I don't care if I can relay phone calls around local WiFi.  I 
don't want a new "Pro" Mac that's expandable only by a rather slow form 
of PCIe.  I'd like to instead get some work done.



> 
> But he is right about HFS+, it's long in the tooth and we need something
> better.
> 
> OTOH, Swift came right out of the blue, so many some next-gen working
> fileOS like zfs but not zfs is on the horizon.

[toc] | [prev] | [next] | [standalone]

#62273

On 2014-07-16, Kevin McMurtrie <mcmurtrie@pixelmemory.us> wrote:
> In article <slrnlsbv5b.1die.g.kreme@Amelia.local>,
>  Lewis <g.kreme@gmail.com.dontsendmecopies> wrote:
>
>> Okay, so one time? In band camp? Jolly Roger <jollyroger@pobox.com> was all, 
>> like: 
>>  --> 	 15 Jul 2014 16:02:52 GMT <c2l1lbFoajbU1@mid.individual.net>
>> > On 2014-07-15, Kevin McMurtrie <mcmurtrie@pixelmemory.us> wrote:
>> >> The Internet net is full of people who claim that Mavericks is much 
>> >> faster or unusably slow.  The truth is - it's both.  Here's some history 
>> >> on what's going on and why you should think twice about investing in 
>> >> more Apple hardware.
>> 
>> > [claims of sluggish performance and stalled apps, along with details
>> > about VM management]
>> 
>> > Meanwhile, I'm seeing no such problems with 10.6, 10.7, 10.8, or 10.9.
>> > Go figure.
>> 
>> McNuttie is a one-note bore.
>
> Apple is the one one-note bore.  They've been reliving the glory of 
> their first iPhone for 7 years now, like a fat middle-aged person still 
> wearing his lucky high school football jacket.  i

Based on your statement, a less knowledgeable person might be misled to
believe that Apple does nothing but sit on their fat asses all day; but
those of us with even the tiniest skill of observation know that is
decidedly not the case, and Apple is heavily innovating, both internally
as well as externally, secret back rooms and released products, on a
fairly regular basis.

> Their desktop systems are dying and iPhones are underwhelming.

Ah. That must be why they sell so well. : )

> I stopped watching Apple's big announcements years ago.  

Was it 7 years ago? Might explain why you think Apple isn't doing much
interesting. ; )

> I don't care about iCloud and it's thimble of storage that isn't
> compatible with anything.  I don't care if I can relay phone calls
> around local WiFi.  I don't want a new "Pro" Mac that's expandable
> only by a rather slow form of PCIe.  I'd like to instead get some work
> done.

While you have harped about your supposed performance issues over the
years, I've happily gotten tons of work done without issue every single
day with my Mac Pro, running each version of Mac OS X that has come out
since I purchased it back in 2008, and it's still *plenty* speedy. The
same can be said for my MacBook Pro and other computers I've owned and
used in that time. Somehow, this problem you claim to have never
appears. It's almost as if...

One-note bore indeed...

-- 
E-mail sent to this address may be devoured by my ravenous SPAM filter.
I often ignore posts from Google. Use a real news client instead.

JR

[toc] | [prev] | [next] | [standalone]

#62294

On Tue, 15 Jul 2014 22:17:34 -0700, Kevin McMurtrie 
<mcmurtrie@pixelmemory.us> wrote:
> In article <slrnlsbv5b.1die.g.kreme@Amelia.local>,
>  Lewis <g.kreme@gmail.com.dontsendmecopies> wrote:
>
>> Okay, so one time? In band camp? Jolly Roger <jollyroger@pobox.com> 
>> was all, like:
>>  --> 15 Jul 2014 16:02:52 GMT <c2l1lbFoajbU1@mid.individual.net>
>> > On 2014-07-15, Kevin McMurtrie <mcmurtrie@pixelmemory.us> wrote:
>> >> The Internet net is full of people who claim that Mavericks is 
>> >> much faster or unusably slow.  The truth is - it's both.  Here's 
>> >> some history on what's going on and why you should think twice 
>> >> about investing in more Apple hardware.
>> 
>> > [claims of sluggish performance and stalled apps, along with 
>> > details about VM management]
>> 
>> > Meanwhile, I'm seeing no such problems with 10.6, 10.7, 10.8, or 
>> > 10.9. Go figure.
>> 
>> McNuttie is a one-note bore.
>
> Apple is the one one-note bore.  They've been reliving the glory of 
> their first iPhone for 7 years now, like a fat middle-aged person 
> still wearing his lucky high school football jacket.  Their desktop 
> systems are dying...

Everybody's desktop systems are dying- Apple, Microsoft, etc.  The 
average smartphone has more computing power than NASA had when they were 
sending astronauts to the moon.  Few people need desktop computers and 
not many need laptops.  Smartphones and tablets are adequate for 
probably 95% of the population.

> and iPhones are underwhelming.

A month or two ago my old cell phone died (after 12 years) and I was 
phone shopping.  Looked at a bunch o' Android and Windows phones and the 
few iPhones that are on the market.  Every one of the Android phones 
were meh for one reason or another- the stupidly monstrous size of many 
of them chief among this.  The iPhone ended up being the most elegant of 
the choices available as well as the easiest to use.  So I got a 5s.  
It's fine, it works, what else do I need.  And on Smart Talk it's $45 a 
month for unlimited talk, text and data (I hate service contracts).

[toc] | [prev] | [next] | [standalone]

#62264

On 2014-07-15 06:38:51 +0200, Kevin McMurtrie said:

> The Internet net is full of people who claim that Mavericks is much
> faster or unusably slow.  The truth is - it's both.  Here's some history
> on what's going on and why you should think twice about investing in
> more Apple hardware.

Nice article. HFS+ is a PITA and the only reason a giant like apple 
could have not to intervene is because they plan to exit the desktop 
market. What else?

I'm trying maczfs under snow leopard. It's wonderful, but it's not 
natively supported. It works better if you have more than one disk, so 
imac are somehow limited in funcionality. But i'm very happy. Once i 
resolved the spotlight limitation (doesn't work on zfs) i will be move 
anything except a little boot partition.

[toc] | [prev] | [next] | [standalone]

#62274

On 2014-07-16, Backdrop <Backdrop> wrote:
> On 2014-07-15 06:38:51 +0200, Kevin McMurtrie said:
>
>> The Internet net is full of people who claim that Mavericks is much
>> faster or unusably slow.  The truth is - it's both.  Here's some history
>> on what's going on and why you should think twice about investing in
>> more Apple hardware.
>
> Nice article. HFS+ is a PITA and the only reason a giant like apple 
> could have not to intervene is because they plan to exit the desktop 
> market. What else?

LOL...

-- 
E-mail sent to this address may be devoured by my ravenous SPAM filter.
I often ignore posts from Google. Use a real news client instead.

JR

[toc] | [prev] | [next] | [standalone]

#62265

On 14-07-15 00:38, Kevin McMurtrie wrote:

> All of this throttling is a very poor workaround to what has been needed 
> for many years: A new filesystem.  HFS+ is too old, 


Is there an objective and technical article that provices a description
of the problems with HFS ?   Is it it due to the "flat" catalogue hidden
outside the file system ? Or due to the way files are created/updated ?

Do other file system really provide significant performance improvements
? I was told that zfs is fast only because it uses a huge amount of RAM
for cache.

Also, are faster/newer file systems more prone to corruption in case of
power failure etc ? Or roughly the same ?

[toc] | [prev] | [next] | [standalone]

#62266

In article <53c5ed7a$0$18235$c3e8da3$3a1a2348@news.astraweb.com>, JF
Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 14-07-15 00:38, Kevin McMurtrie wrote:
> > All of this throttling is a very poor workaround to what has been needed 
> > for many years: A new filesystem.  HFS+ is too old, 
> 
> 
> Is there an objective and technical article that provices a description
> of the problems with HFS ?   Is it it due to the "flat" catalogue hidden
> outside the file system ? Or due to the way files are created/updated ?

there aren't any significant problems with hfs+.

he's full of shit.

[toc] | [prev] | [next] | [standalone]

#62268

Okay, so one time? In band camp? JF Mezei <jfmezei.spamnot@vaxination.ca> was all, like: 
 --> 	 Tue, 15 Jul 2014 23:11:54 -0400 <53c5ed7a$0$18235$c3e8da3$3a1a2348@news.astraweb.com>
> On 14-07-15 00:38, Kevin McMurtrie wrote:

>> All of this throttling is a very poor workaround to what has been needed 
>> for many years: A new filesystem.  HFS+ is too old, 


> Is there an objective and technical article that provices a description
> of the problems with HFS ?   Is it it due to the "flat" catalogue hidden
> outside the file system ? Or due to the way files are created/updated ?

This is pretty thorough:

<http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/#hfs-problems>
"When HFS was created, the top-of-the-line Macintosh came with an 800K
floppy drive, the "high-end" storage offered by Apple was a 20MB hard
drive the size of a lunchbox, and the CPU was from the Motorola 68000
family. Thirteen years later, HFS+ replaced HFS, the floppy disks were
1.44MB, and Apple's hard drives topped out around 6GB."

> Do other file system really provide significant performance improvements

Significant improvements, yes. Reliability and data integrity in
particular. Some performance, but that's not the real issue.

> Also, are faster/newer file systems more prone to corruption in case of
> power failure etc ? Or roughly the same ?

Much much much less. A modern file system does many things to protect
and verify data integrity where HFS+ does precisely zero. When you write
a file to HFS+ the assumption is that the file was written correctly, no
checking is done.


-- 
"We're philosophers. We think, therefore we am."

[toc] | [prev] | [next] | [standalone]

#62271

On 14-07-16 00:31, Lewis wrote:

> This is pretty thorough:
> 
> <http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/#hfs-problems>


I'd love to see an Apple response to this, to get the "other side of the
coin".

Interesting that VMS, older than HFS, stores precice time values (64
bits, nanoseconds since 1858) and of course supports proper locking of
files.

However, in fairness, most Unixes seem to drop OS level locks because
databases are now the ones who provide record level locks, and
sophisiticated databases can lock across nodes (something which VMS does
natively at the OS level).

The Advanced File system for Digital Unix/Tru64 was apparently very well
perceived and at the top of various Unix file systems (despite not being
popular in market due to Digital not marketing its product).

BTW, didn't HFS come out with the Mac+ which has 1.44 floppies ? The
original Mac had a emulated folders with a flat file and 800k floppies
as I recall.

I am surprised that the file system would have remained big endian.
Figured Apple would have created a little endian version of HFS+ (a flag
on teh disk would trigger the little endian version of the HFS file system).

What the Ars did not discuss however is whether the general structure of
the HFS+ is good or not. Yeah, hard links in a hidden directrory, but
they fail to mention that the real directory/catalogue for all files is
outside the file system. And I'd like to know if this is still a va;id
concept these days from a performance/scalability point of view.

VMS had directory files that contained list of files in them, but each
file had a numeric ID, and the "real" file was catalogued in the root
level INDEXF.SYS file where each's files metadata (dates, size, extents,
protection, owner etc) was stored.

[toc] | [prev] | [next] | [standalone]

#62275

JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 14-07-16 00:31, Lewis wrote:
> 
> > This is pretty thorough:
> > 
> > <http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/#hfs-problems>
> 
> 
> I'd love to see an Apple response to this, to get the "other side of the
> coin".
> 
> Interesting that VMS, older than HFS, stores precice time values (64
> bits, nanoseconds since 1858) and of course supports proper locking of
> files.
> 
> However, in fairness, most Unixes seem to drop OS level locks because
> databases are now the ones who provide record level locks, and
> sophisiticated databases can lock across nodes (something which VMS does
> natively at the OS level).
> 
> The Advanced File system for Digital Unix/Tru64 was apparently very well
> perceived and at the top of various Unix file systems (despite not being
> popular in market due to Digital not marketing its product).
> 
> BTW, didn't HFS come out with the Mac+ which has 1.44 floppies ?

HFS did come with the Mac Plus (January 1986), and it was with the 800K
floppy, as the article said.

1.44 MB floppy drives were introduced on the Mac IIx (September 1988),
and could be retrofitted as an upgrade (drive and controller) on the Mac
II and Mac SE. There was also an external 1.44 MB floppy drive, called
the "SuperDrive" (that name was since reused at least once to refer to
an optical drive which can write DVDs). I still have my 1.44 MB external
floppy SuperDrive, along with the controller card in my Apple IIgs.

> The original Mac had a emulated folders with a flat file and 800k floppies
> as I recall.

Yes to the emulated folders. After HFS was introduced, the original file
system was known as MFS for "Macintosh File System". Folders were
implemented by a field in the flat directory which contained a folder ID
number. Finder displayed it as a directory hierarchy. HFS had a genuine
hierarchical directory.

The original Mac (January 1984) had a 400K (single sided) 3.5" floppy,
which had previously been used on a revision to the Lisa.

The 800K (double sided) floppy drive came with the Mac Plus, and a few
months later to the 512Ke (not sure whether the Mac 512K could also be
upgraded to an 800K floppy drive and 128K ROM, as I wasn't paying as
much attention to Macs prior to the SE).

> I am surprised that the file system would have remained big endian.
> Figured Apple would have created a little endian version of HFS+ (a flag
> on teh disk would trigger the little endian version of the HFS file system).

Why bother? The endianness has a very minor impact on CPU performance
for Intel Macs, which is hardly an issue compared to disk I/O or other
performance bottlenecks in HFS+ (e.g. single threaded file system).
Changing this would have forced a compatibility break with older
versions of Mac OS X, for no significant gain, and with the potential of
new bugs.

Intel Macs already need to deal with big-endian data being standard in
network protocols. Big-endian fields are also easier for humans to read
in raw data dumps. x86 processors used by Apple have a BSWAP instruction
which reverses the bytes in a 32-bit word.

The only point I can see where it might have been reasonable to make
this change was when Apple added support for GUID partition table (Mac
OS X 10.4), since that also forced a compatibility break with Mac OS X
10.3.9 and earlier (only affecting PowerPC or 68K Macs accessing a drive
partitioned using Mac OS 10.4 or later for an Intel Mac).

Changing endianness in HFS+ (with a new file system type) at any other
point would have forced a compatibility break for a probably
unmeasurable performance change.

> What the Ars did not discuss however is whether the general structure of
> the HFS+ is good or not. Yeah, hard links in a hidden directrory, but
> they fail to mention that the real directory/catalogue for all files is
> outside the file system.

What are you talking about? The directory (catalog file and several
other associated files) is implemented using files within the file
system in HFS+.

https://developer.apple.com/legacy/library/technotes/tn/tn1150.html#//apple_ref/doc/uid/DTS10002989

This says it hasn't been updated since 2004 (though the index says it
was updated in 2011) so it is missing details like hard links to
directories (used in Leopard and later for Time Machine). I seem to
remember reading about that somewhere, perhaps in a later technical
note, but can't lay my hands on it at the moment.

> And I'd like to know if this is still a valid concept these days from a
> performance/scalability point of view.
> 
> VMS had directory files that contained list of files in them, but each
> file had a numeric ID, and the "real" file was catalogued in the root
> level INDEXF.SYS file where each's files metadata (dates, size, extents,
> protection, owner etc) was stored.

-- 
David Empson
dempson@actrix.gen.nz

[toc] | [prev] | [next] | [standalone]

#62278

On 14-07-16 03:39, David Empson wrote:

> What are you talking about? The directory (catalog file and several
> other associated files) is implemented using files within the file
> system in HFS+.
> 
> https://developer.apple.com/legacy/library/technotes/tn/tn1150.html#//apple_ref/doc/uid/DTS10002989

So what is the file name for the catalogue file ?

I've started to read the above, but missing something important.

Catalogue file is a B tree with numbers as keys (fileID, directory ID
for instance).

So what is do ls /Users/jfmezei/

what actually happens ?  I have to assume that it doesn't go through the
catalogue sequentially to find all files whose parent directory is the
DIRID associated with jfmezei

looks to me like the catyalogue file is very much like INDEXF.SYS , the
master list of files and which disk blocks contain their data. But to
get to it, you need the file ID.

So how does it go from filename to file ID ?

I had been told that there are no directory files. So Is there somewhere
that stores the list of files in a particular directory ?


> This says it hasn't been updated since 2004 (though the index says it
> was updated in 2011) so it is missing details like hard links

Are hard links to directories different than hardlinks to files ?
(since I don't yet understand how directories are implemented in HFS+,
it is hard for me to understand the difference).

It is interesting that Apple was able to update HFS to HFS+, change some
values from 16 to 32 bits. So why not HFS++ where it could do little
endian, and increase other values to 64 bits ?

[toc] | [prev] | [next] | [standalone]

#62298

JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 14-07-16 03:39, David Empson wrote:
> 
> > What are you talking about? The directory (catalog file and several
> > other associated files) is implemented using files within the file
> > system in HFS+.
> > 
> > https://developer.apple.com/legacy/library/technotes/tn/tn1150.html#//ap
> > ple_ref/doc/uid/DTS10002989
> 
> So what is the file name for the catalogue file ?

It doesn't have a filename, nor do the other special files that manage
the file system (extents overflow file, attributes file, allocation file
and startup file).

It is a while since I've looked at this in detail, and I don't have time
right now to re-read the technote in full or have a look at an actual
disk to confirm details.

The catalog file and other special files are allocated in the same way
as normal files. They differ in that the start of them is located via
fixed entries in the volume header, rather than having to search for
them in the catalog file. I don't recall offhand whether they even have
an entry in the catalog file, but they can appear in the extents
overflow file.

> I've started to read the above, but missing something important.
> 
> Catalogue file is a B tree with numbers as keys (fileID, directory ID
> for instance).
> 
> So what is do ls /Users/jfmezei/
> 
> what actually happens ?  I have to assume that it doesn't go through the
> catalogue sequentially to find all files whose parent directory is the
> DIRID associated with jfmezei

If all you had was the pathname, that would involve two searches of the
catalog file B-Tree:

1. Given the folder ID of the root directory (a fixed constant), find an
item called Users with that parent ID. Confirm it is itself a directory
and fetch its folder ID.

2. Given the folder ID of /Users, find an item called jfmezei with the
parent ID maching the folder ID of /Users.

Neither of these searches is sequential: the catalog tree entries for
file and folder names are indexed by the B-Tree with (parentID,
filename) as the sort key, so it is an indexed search to locate the
appropriate entry.

> looks to me like the catyalogue file is very much like INDEXF.SYS , the
> master list of files and which disk blocks contain their data. But to
> get to it, you need the file ID.
> 
> So how does it go from filename to file ID ?

If you are working with a full pathname, you recursively locate each
folder in the path, starting from the root directory, given the known
parentID at each point.

Applications which use Apple's native file APIs (as opposed to the Unix
APIs) don't require this much work, because they aren't usually dealing
with full pathnames: they have an alias record to locate a file (or
folder), and the alias record holds the fileID, parentID and filename. 

For example, if you use Pages to open a document, Pages is given the
volume-wide unique fileID of the document to open. This requires one
indexed lookup in the catalog tree to locate a file thread record with
that fileID, which is used to fetch the current parentID and filename.
Another indexed lookup then locates the file.

If the file had been deleted and recreated in place, the fileID in the
alias record would no longer be valid, so after the fileID search
failed, the cached parentID and filename would fetched from the alias
record, allowing the file to be located again.

If the file had been moved to a different folder, or renamed, or its
parent folder had been renamed or moved, then the fileID is still valid
and the file would still be located.

This is the basis of how alias files work, and how they are able to
follow a file or folder when you move the original: the fileID is not
altered.

If a file is both replaced and moved/renamed, then the alias won't be
able to find it.

> I had been told that there are no directory files. So Is there somewhere
> that stores the list of files in a particular directory ?

The entire directory for the volume is stored in the single catalog
file, allowing all files on the volume to be located either by
(parentID, filename) or by fileID/folderID.

The files within a particular directory can be located by finding all
entries with the same parentID, which will be logically adjacent in the
catalog file because the parentID/folderID/fileID is the primary sort
key of the catalog file B-Tree.

> > This says it hasn't been updated since 2004 (though the index says it
> > was updated in 2011) so it is missing details like hard links
> 
> Are hard links to directories different than hardlinks to files ?
> (since I don't yet understand how directories are implemented in HFS+,
> it is hard for me to understand the difference).

That document says hard links to directories are not allowed (because it
opens up the possibility of a circular path in the directory). That rule
was relaxed in Leopard, but only for usage like Time Machine, which
creates hard links to directories at the same level of different
branches of the logical directory tree.

Hard links to files in HFS+ are implemented via indirect node files
stored in the hidden "HFS+ Private Data" folder in the root directory. I
expect hard links to directories are implemented in the same way, just
with the target of the link being a folder rather than a file.

I came across this blog page which has some details, but haven't had
time to read it in full. (The referenced hfsdebug software is no longer
available, and was never updated to be Intel native.)

http://osxbook.com/blog/2008/11/09/hfsdebug-40-and-new-hfs-features/

> It is interesting that Apple was able to update HFS to HFS+, change some
> values from 16 to 32 bits. So why not HFS++ where it could do little
> endian, and increase other values to 64 bits ?

There is no point going to 64 bit fields until there is a need for more
than 4 billion files on a single logical volume, or once volume sizes
get large enough that the allocation blocks are having to grow too much.
That isn't likely to be an issue for consumers or small businesses any
time soon, and Apple is no longer interested in trying to implement
enterprise servers.

HFS+ can use 4 KiB allocation blocks for volume sixes up to about 16
TiB. Beyond that point it can use larger allocation blocks, which isn't
going to matter much if the number of files on the volume is still
several orders of magnitude less than the number of allocation blocks.

The OS might choose to use larger allocation blocks on somewhat smaller
volumes (e.g. above 1 TiB), to keep the total number of allocation
blocks down to a more manageable number. I haven't investigated whether
that is happening already.

HFS was running into the problem with increasingly large allocation
blocks around the 1 GiB volume size, and also had a hard limit of about
65K files per volume.

I hope that by the time 16 TB consumer drives are available, Apple will
have a new file system anyway.

-- 
David Empson
dempson@actrix.gen.nz

[toc] | [prev] | [next] | [standalone]

#62299

On 14-07-17 21:06, David Empson wrote:

> It doesn't have a filename, nor do the other special files that manage
> the file system (extents overflow file, attributes file, allocation file
> and startup file).

So they may be part of the volume, but not part of the file system. But
I guess this gets into semantics.

> 1. Given the folder ID of the root directory (a fixed constant), find an
> item called Users with that parent ID. Confirm it is itself a directory
> and fetch its folder ID.

> Neither of these searches is sequential: the catalog tree entries for
> file and folder names are indexed by the B-Tree with (parentID,
> filename) as the sort key, so it is an indexed search to locate the
> appropriate entry.

So the primary and only key into the catalogue file is numerid parentID
combined with text filename ?

[toc] | [prev] | [next] | [standalone]

#62301

JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 14-07-17 21:06, David Empson wrote:
> 
> > It doesn't have a filename, nor do the other special files that manage
> > the file system (extents overflow file, attributes file, allocation file
> > and startup file).
> 
> So they may be part of the volume, but not part of the file system. But
> I guess this gets into semantics.

It depends on your definition. They are allocated from the same pool of
allocation blocks as files, and are organised into extents like other
files. The differences are that they are not children of the root
directory, and are located via fixed entries in the volume header rather
than by looking up a catalog entry.

I've now found the bit mentioning in more detail how the special files
are handled: the volume header has an HFSPlusForkData structure for each
of the special files, which includes the logical size, clump size, total
blocks and array of eight extent descriptors.

If more than eight extents are needed, the additional records are in the
extents overflow file (as is the case for a fork of a normal file).

That means the special files don't have an entry in the catalog file, so
they don't have attributes such as permissions, modification time, etc.

> > 1. Given the folder ID of the root directory (a fixed constant), find an
> > item called Users with that parent ID. Confirm it is itself a directory
> > and fetch its folder ID.
> 
> > Neither of these searches is sequential: the catalog tree entries for
> > file and folder names are indexed by the B-Tree with (parentID,
> > filename) as the sort key, so it is an indexed search to locate the
> > appropriate entry.
> 
> So the primary and only key into the catalogue file is numerid parentID
> combined with text filename ?

For file and folder records, yes. That is the HFSPlusCatalogKey
structure. The leaf node (data) for these records is the catalog entry
for the file or folder (e.g. including its modification time, access
permissions, location of its first extent, etc.).

File and folder thread records (used to locate a file or folder given
its fileID or folderID) have a compatible but slightly different key:
the same data structure is used but the parentID field holds the fileID
or folderID (which are the same numbering system as parentID), and the
filename field is an empty string. The leaf node (data) for the thread
record contains the parentID and filename.

If the OS is searching for a file or folder given a known parentID and
filename (or iterating through the items in a folder), it will be
looking for file/folder records, and this yields the catalog entry.

If the OS is searching for a file or folder's catalog entry given a
fileID or folderID, it will be looking for a specific file/folder thread
record. This yields the parentID and filename. A second lookup is
required to find the file/folder record given the parentID and filename,
which then yields the catalog entry.

In the case of a folder, the sort order and the empty filename field
means that the thread record for a folder is immediately followed (in
logical order) by the file/folder records for all items in that folder,
since all those records have the same value in the parentID field, and
the empty string sorts before the first possible filename.

-- 
David Empson
dempson@actrix.gen.nz

[toc] | [prev] | [next] | [standalone]

#62277

Okay, so one time? In band camp? JF Mezei <jfmezei.spamnot@vaxination.ca> was all, like: 
 --> 	 Wed, 16 Jul 2014 01:06:15 -0400 <53c60847$0$1663$c3e8da3$5d8fb80f@news.astraweb.com>
> What the Ars did not discuss however is whether the general structure of
> the HFS+ is good or not.

It's not terrible, but it's not good. Some of that WAS covered (for
example, the 16 bit limits on searching the btree).

Well, Siracusa would say it was terrible, but he starts at "bad" and
moves down. :)

-- 
I WAS NOT TOUCHED "THERE" BY AN ANGEL Bart chalkboard Ep. BABF14

[toc] | [prev] | [next] | [standalone]

#62276

On 2014.07.15, 00:38 , Kevin McMurtrie wrote:
> The Internet net is full of people who claim that Mavericks is much
> faster or unusably slow.  The truth is - it's both.  Here's some history
> on what's going on and why you should think twice about investing in
> more Apple hardware.

The only issue I've had is on an older Mac Book Air with some apps, at 
some times.  Of late there have been no issues at all on the MBA.

Nor on the iMac (2007) that I sold, MBP, this iMac or the new MBA 
acquired a few weeks ago.

Not saying that is proof, but I don't think it's a general problem as 
much as problems with certain machines which might have other issues.

I admit that the throttling seems indiscriminate in its application - 
OTOH I have no evidence that it actually interferes with the general use 
of my computers.  I don't have many 'disk intensive' apps (other than 
one I wrote myself that still manages to treesearch the entire volume 
and process thousands of GPS recordings within a minute or 2.)

-- 
I was born a 1%er - I'm just more equal than the rest.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.sys.mac.system

csiph-web