Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.sys.mac.system > #119579

Re: APFS on HDD

Show all headers | View raw

On 2018-11-18 00:37, David Empson wrote:

> That may have been the case in APFS version 1 (developer preview on
> Sierra), which Apple hasn't documented in detail. I haven't tried
> multi-volume boot scenerios yet, except in separate
> partitions/containers.

When I watched the APFS presentation in 2017, it mentiond that APFS
basically placed the boot information (block location, number of blocks
to load etc) in block 0 of the GPT partition where the EFI firmware
would look to see if the GPT partition was bootable.

So from an EFI point of view, it could only see one bootable volume in
the GPT partition that containes the APFS container which may contain
multiple bootable APFS volumes)

Considering the recovery partition is also bootable and exists in the
same APFS container, this would likekly require that a neutral "pre
boot" environment is what EFI actually loads and that pre-boot then
decides which APFS volume inside the APFS container is to be loaded.
(the pre-boot stuff is what the EFI partition was actually meant to do,
as well as be the one which understands how to parse the file system
containing the boot volume to load the primary bootstrap file.

And with encryption, a pre-boot middleware between EFO and actuao boot
would be required so that something can ask for password needed to
decrupt the boot drive before being able to load the primary bootstrap
for the OS.


But that would mean that at the EFI level, it cannot list all bootable
volumes, as it only sees bootable GPT partitions. (unless Apple adds
APFS understanding to the EFI firmware inside its computers)



>> "The container layer organizes file-system layer information and stores
>> higher level information, such as volume metadata, snapshots of the
>> volume, and encryption state"
>> 
>> This is interesting because later on, the VOLUME header contains pointer
>> to a B-Tree list of snapshots.
> 
> I think the quoted sentence is in error about snapshots. I can't see any
> other evidence of snapshot information being outside the scope of a
> volume.


Is it possible that space used by the snapshot (keeping certain blocks
"alive" when the source volume has released them) gets "charged" to the
snapshot at the APFS container level instead of being considered as part
of disk blocks used by the volume ? (which  would then not jive with
totaal blcoks used by files in that volume)



> That is what "tmutil listlocalsnapshots /" shows, though that list may
> be filtered to only list snapshots created by Time Machine and related
> tools, not all current snapshots created directly through the API.

list local snapshots gives names of snapshots in pretty output.
Consider a scenario where snapthots were indexed in the container for
all APFS volumes in that container.  Listlocalsnapshots / would simply
list those snamshots that belong to the current device that is the file
sytstem root. The output of the utility would be identical, even though
th structure deep down would be very different.



> Yes. The volume superblock has an apfs_root_tree_oid field which points
> (indirectly) to the root of a B-tree which contains the entire
> filesystem of the volume.

Yeah, I ended up making that conclusion, but I assume the document just
assumed everyone knew that the concept of a catalogue file from HFS was
maintained.


> Its internal structure and indexing arangement is very different to
> HFS+. Changes to the filesystem only require copy-on-write replacement
> of modified blocks of the B-tree.

In VMS for instance, the equivalemt of the catalogue file was INDEXF.SYS
(but only had file entries, not directory hiearchy information), and
each record had a fixed amount of space for extents, and when a file
became badly fragmented, an additional "continuation" record was created
to contain the additional extents.

Considering how APFS has gone "extent crazy" potentially resulting large
files with 1 extent per block, I am curious how they designed the
records in the big Btree catalogue to handle large number of extents.

I take it a snapshot has its own btree catalogue of the files that it
contains? or does it share use of the volume's catalogue B-tree ?

> For those aware of the horrible hack HFS+ had to use to implement hard
> links, a major point is that the APFS file system tree has the inode
> number as its primary index,

Yeah, it appears that they kept the same overall architecture for the
file system but added the fields needed to make it far more efficient.

> Unlike HFS+, there is no attempt to internally group file records by
> their parent directory. 

is the file AAAA.readme.txt stored anywhere in the b-tree (with primary
key being inode) or is some alphabetical order maintained in the b-tree?

In other words would
/etc/bind/myconfig/AAAA.readme    and
/System/Library/launchdaemons/documentation/AAAA.readme

be near each other in the B-tree or be very far apart ?


> I haven't looked closely at the details of extents yet but in principle
> the defragmenter just needs to move file extents to be in adjacent
> physical blocks, 

But when I update block 75 or a 200 block file, block 75 get written far
away and the file then becomes an extent for 0-74, one for 75 and one
for 76-200.

The problem is that a snapshop would retain single 0-200 extent with its
block 75 containing the old informaion.

When a defragmenter wants to recreate the live file to have one
conti8guious extent, it would move blocks 0-74 and 76-200 somewhere
else.  It would likely then have to update the snapshot to have blocks
0-74 poiting to the new location of the defragmented live file, block 74
pointing to the old location and blcosk 76-200 to the new location. (in
essence, the snapshot becomes fragmented in order to defragment the live
file. So when you have many snapshots, this can become inwieldy when you
often update random blocks in a file.

I was really expecting/hoping for an APFS update at this year WWDC to
discuss how they tackled that challenge.

Back to comp.sys.mac.system | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: APFS on HDD rs99tl@gmail.com - 2018-11-09 17:32 -0800
  Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-10 14:42 +1300
    Re: APFS on HDD RSORLANDO <rs99tl@gmail.com> - 2018-11-09 18:15 -0800
    Re: APFS on HDD Elden <usenet@moondog.org> - 2018-11-09 21:27 -0800
      Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-10 18:52 +1300
        Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-10 09:11 -0500
          Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-11 10:08 +1300
            Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-10 17:41 -0500
              Re: APFS on HDD nospam <nospam@nospam.invalid> - 2018-11-10 19:18 -0500
              Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-11 18:03 +1300
                Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-11 13:21 -0500
                Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-12 03:09 +0000
                Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-12 02:10 -0500
                Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-12 12:32 +0000
                Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-13 08:14 +1300
                Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-12 20:29 -0500
                Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-13 18:15 +1300
                Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-13 09:20 -0500
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 02:30 -0500
                Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-12 12:45 +0000
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 18:31 -0500
                Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-13 00:04 +0000
                Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-13 13:46 +1300
          Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-11 15:55 +1300
            Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-10 23:38 -0500
            Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-11 07:18 +0000
              Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-11 12:39 -0500
                Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-12 09:10 +1300
                Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-11 19:26 -0500
                Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-12 16:20 +1300
                Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-12 01:46 -0500
            Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-14 02:05 +1300
        Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-11 07:13 +0000
          Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 02:10 -0500
        Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 02:02 -0500
          Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-12 21:57 +1300
            Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 18:26 -0500
              Re: APFS on HDD Jolly Roger <jollyroger@pobox.com> - 2018-11-13 01:12 +0000
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 23:44 -0500
                Re: APFS on HDD Jolly Roger <jollyroger@pobox.com> - 2018-11-13 18:17 +0000
              Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-13 19:38 +1300
                Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-13 10:21 +0000
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-13 05:55 -0500
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-14 01:28 +1300
                Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-13 13:40 +0000
                Re: APFS on HDD Jolly Roger <jollyroger@pobox.com> - 2018-11-13 18:18 +0000
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-13 05:57 -0500
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-14 01:28 +1300
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-13 13:27 -0500
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-14 13:31 +1300
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-14 15:33 -0500
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-15 12:04 +1300
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-15 00:35 -0500
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-15 22:24 +1300
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-15 12:39 -0500
                Re: APFS on HDD Stephen Hoffman <seaohveh@hoffmanlabs.invalid> - 2018-11-16 13:45 -0500
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-16 20:13 -0500
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-18 18:37 +1300
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-18 17:40 -0500
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-19 13:42 +1300
                Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-19 02:08 +0000
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-19 15:52 +1300
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-18 22:31 -0500
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-20 13:23 +1300
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-19 23:32 -0500
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-20 22:05 +1300
                Re: APFS on HDD "David B." <"David B"@nomail.afraid.org> - 2018-11-20 09:36 +0000
                Re: APFS on HDD André G. Isaak <agisaak@gm.invalid> - 2018-11-20 06:12 -0700
                Re: APFS on HDD "David B." <"David B"@nomail.afraid.org> - 2018-11-20 14:28 +0000
                Re: APFS on HDD nospam <nospam@nospam.invalid> - 2018-11-20 09:36 -0500
                Re: APFS on HDD nospam <nospam@nospam.invalid> - 2018-11-20 08:57 -0500
                Re: APFS on HDD "David B." <"David B"@nomail.afraid.org> - 2018-11-20 14:34 +0000
                Re: APFS on HDD nospam <nospam@nospam.invalid> - 2018-11-20 09:36 -0500
                Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-19 05:46 +0000
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-19 02:27 -0500
                Re: APFS on HDD Stephen Hoffman <seaohveh@hoffmanlabs.invalid> - 2018-11-19 11:42 -0500
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-20 13:23 +1300
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-19 21:57 -0500
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-20 16:28 +1300
                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-21 13:24 -0500
                Re: APFS on HDD Stephen Hoffman <seaohveh@hoffmanlabs.invalid> - 2018-11-19 11:34 -0500
                Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-13 13:41 +0000
  Re: APFS on HDD Alan Browne <bitbucket@blackhole.com> - 2018-11-11 13:04 -0500

csiph-web