Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.sys.mac.system > #119404 > unrolled thread

Re: APFS on HDD

Back to article view | Back to comp.sys.mac.system

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: APFS on HDD rs99tl@gmail.com - 2018-11-09 17:32 -0800
    Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-10 14:42 +1300
      Re: APFS on HDD RSORLANDO <rs99tl@gmail.com> - 2018-11-09 18:15 -0800
      Re: APFS on HDD Elden <usenet@moondog.org> - 2018-11-09 21:27 -0800
        Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-10 18:52 +1300
          Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-10 09:11 -0500
            Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-11 10:08 +1300
              Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-10 17:41 -0500
                Re: APFS on HDD nospam <nospam@nospam.invalid> - 2018-11-10 19:18 -0500
                Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-11 18:03 +1300
                  Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-11 13:21 -0500
                    Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-12 03:09 +0000
                      Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-12 02:10 -0500
                        Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-12 12:32 +0000
                        Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-13 08:14 +1300
                          Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-12 20:29 -0500
                            Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-13 18:15 +1300
                              Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-13 09:20 -0500
                      Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 02:30 -0500
                        Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-12 12:45 +0000
                          Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 18:31 -0500
                            Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-13 00:04 +0000
                            Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-13 13:46 +1300
            Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-11 15:55 +1300
              Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-10 23:38 -0500
              Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-11 07:18 +0000
                Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-11 12:39 -0500
                  Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-12 09:10 +1300
                    Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-11 19:26 -0500
                      Re: APFS on HDD Your Name <YourName@YourISP.com> - 2018-11-12 16:20 +1300
                        Re: APFS on HDD J Burns <burns@nospam.com> - 2018-11-12 01:46 -0500
              Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-14 02:05 +1300
          Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-11 07:13 +0000
            Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 02:10 -0500
          Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 02:02 -0500
            Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-12 21:57 +1300
              Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 18:26 -0500
                Re: APFS on HDD Jolly Roger <jollyroger@pobox.com> - 2018-11-13 01:12 +0000
                  Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-12 23:44 -0500
                    Re: APFS on HDD Jolly Roger <jollyroger@pobox.com> - 2018-11-13 18:17 +0000
                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-13 19:38 +1300
                  Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-13 10:21 +0000
                  Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-13 05:55 -0500
                    Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-14 01:28 +1300
                    Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-13 13:40 +0000
                      Re: APFS on HDD Jolly Roger <jollyroger@pobox.com> - 2018-11-13 18:18 +0000
                  Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-13 05:57 -0500
                    Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-14 01:28 +1300
                      Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-13 13:27 -0500
                        Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-14 13:31 +1300
                          Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-14 15:33 -0500
                            Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-15 12:04 +1300
                              Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-15 00:35 -0500
                                Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-15 22:24 +1300
                                  Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-15 12:39 -0500
                                    Re: APFS on HDD Stephen Hoffman <seaohveh@hoffmanlabs.invalid> - 2018-11-16 13:45 -0500
                                      Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-16 20:13 -0500
                                        Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-18 18:37 +1300
                                          Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-18 17:40 -0500
                                            Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-19 13:42 +1300
                                            Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-19 02:08 +0000
                                              Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-19 15:52 +1300
                                                Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-18 22:31 -0500
                                                  Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-20 13:23 +1300
                                                    Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-19 23:32 -0500
                                                      Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-20 22:05 +1300
                                                        Re: APFS on HDD "David B." <"David B"@nomail.afraid.org> - 2018-11-20 09:36 +0000
                                                          Re: APFS on HDD André G. Isaak <agisaak@gm.invalid> - 2018-11-20 06:12 -0700
                                                            Re: APFS on HDD "David B." <"David B"@nomail.afraid.org> - 2018-11-20 14:28 +0000
                                                              Re: APFS on HDD nospam <nospam@nospam.invalid> - 2018-11-20 09:36 -0500
                                                          Re: APFS on HDD nospam <nospam@nospam.invalid> - 2018-11-20 08:57 -0500
                                                            Re: APFS on HDD "David B." <"David B"@nomail.afraid.org> - 2018-11-20 14:34 +0000
                                                              Re: APFS on HDD nospam <nospam@nospam.invalid> - 2018-11-20 09:36 -0500
                                                Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-19 05:46 +0000
                                                  Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-19 02:27 -0500
                                                    Re: APFS on HDD Stephen Hoffman <seaohveh@hoffmanlabs.invalid> - 2018-11-19 11:42 -0500
                                                  Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-20 13:23 +1300
                                                    Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-19 21:57 -0500
                                                      Re: APFS on HDD dempson@actrix.gen.nz (David Empson) - 2018-11-20 16:28 +1300
                                                        Re: APFS on HDD JF Mezei <jfmezei.spamnot@vaxination.ca> - 2018-11-21 13:24 -0500
                                            Re: APFS on HDD Stephen Hoffman <seaohveh@hoffmanlabs.invalid> - 2018-11-19 11:34 -0500
                    Re: APFS on HDD Lewis <g.kreme@gmail.com.dontsendmecopies> - 2018-11-13 13:41 +0000
    Re: APFS on HDD Alan Browne <bitbucket@blackhole.com> - 2018-11-11 13:04 -0500

Page 3 of 5 — ← Prev page 1 2 [3] 4 5  Next page →

#119515

JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 2018-11-12 03:57, David Empson wrote:
> 
> > That is exactly how TM already does backups from a source APFS volume
> > (on High Sierra and later). Snapshots aren't deleted after doing a
> > backup to an external drive - that snapshot doubles as another local
> > snapshot.
> 
> Since my hard drive is APFS, how do I go about finding the snapshot made
> whenever Time Machine makes a backup ?

man tmutil

Find the word "snapshot" in the man page.

To be precise:

tmutil listlocalsnapshots /

(as has been mentioned by many people in this group many times).

It doesn't tell you which one was used to make a TM backup. Some of them
are just local snapshots, created automatically and periodically. You'd
need to compare with your backup drive.

> Or it it made is some hidden APFS partition/volume?

That wouldn't work. Snapshots must be on the volume they belong to.

-- 
David Empson
dempson@actrix.gen.nz

[toc] | [prev] | [next] | [standalone]

#119518

In message <1ny7lc8.19eb80bgrvzdmN%dempson@actrix.gen.nz> David Empson <dempson@actrix.gen.nz> wrote:
> JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

>> On 2018-11-12 03:57, David Empson wrote:
>> 
>> > That is exactly how TM already does backups from a source APFS volume
>> > (on High Sierra and later). Snapshots aren't deleted after doing a
>> > backup to an external drive - that snapshot doubles as another local
>> > snapshot.
>> 
>> Since my hard drive is APFS, how do I go about finding the snapshot made
>> whenever Time Machine makes a backup ?

> man tmutil

> Find the word "snapshot" in the man page.

> To be precise:

> tmutil listlocalsnapshots /

> (as has been mentioned by many people in this group many times).

And trivially findable on your favorite search engine, a skill that JF
often lacks.

> It doesn't tell you which one was used to make a TM backup. Some of them
> are just local snapshots, created automatically and periodically. You'd
> need to compare with your backup drive.

>> Or it it made is some hidden APFS partition/volume?

> That wouldn't work. Snapshots must be on the volume they belong to.

Just like files sharing space.

-- 
Everywhere is walking distance if you have the time.

[toc] | [prev] | [next] | [standalone]

#119519

On 2018-11-13 01:38, David Empson wrote:
> tmutil listlocalsnapshots


I get
bike:~ $  tmutil listlocalsnapshots /
com.apple.TimeMachine.2018-11-12-060805
com.apple.TimeMachine.2018-11-12-070802
com.apple.TimeMachine.2018-11-12-081408
com.apple.TimeMachine.2018-11-12-091017
com.apple.TimeMachine.2018-11-12-101438
com.apple.TimeMachine.2018-11-12-111209
com.apple.TimeMachine.2018-11-12-130716
com.apple.TimeMachine.2018-11-12-140815
com.apple.TimeMachine.2018-11-12-150756
com.apple.TimeMachine.2018-11-12-161008
com.apple.TimeMachine.2018-11-12-171211
com.apple.TimeMachine.2018-11-12-181609
com.apple.TimeMachine.2018-11-12-200722
com.apple.TimeMachine.2018-11-12-211156
com.apple.TimeMachine.2018-11-12-220722
com.apple.TimeMachine.2018-11-12-231216
com.apple.TimeMachine.2018-11-13-001553
com.apple.TimeMachine.2018-11-13-011156
com.apple.TimeMachine.2018-11-13-020721
com.apple.TimeMachine.2018-11-13-040721
com.apple.TimeMachine.2018-11-13-051250


So where would com.apple.TimeMachine.2018-11-12-060805  be located in
the boot disk? "locate" doesn't find it.

[toc] | [prev] | [next] | [standalone]

#119522

JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 2018-11-13 01:38, David Empson wrote:
> > tmutil listlocalsnapshots
> 
> 
> I get
> bike:~ $  tmutil listlocalsnapshots /
> com.apple.TimeMachine.2018-11-12-060805
> com.apple.TimeMachine.2018-11-12-070802
> com.apple.TimeMachine.2018-11-12-081408
> com.apple.TimeMachine.2018-11-12-091017
> com.apple.TimeMachine.2018-11-12-101438
> com.apple.TimeMachine.2018-11-12-111209
> com.apple.TimeMachine.2018-11-12-130716
> com.apple.TimeMachine.2018-11-12-140815
> com.apple.TimeMachine.2018-11-12-150756
> com.apple.TimeMachine.2018-11-12-161008
> com.apple.TimeMachine.2018-11-12-171211
> com.apple.TimeMachine.2018-11-12-181609
> com.apple.TimeMachine.2018-11-12-200722
> com.apple.TimeMachine.2018-11-12-211156
> com.apple.TimeMachine.2018-11-12-220722
> com.apple.TimeMachine.2018-11-12-231216
> com.apple.TimeMachine.2018-11-13-001553
> com.apple.TimeMachine.2018-11-13-011156
> com.apple.TimeMachine.2018-11-13-020721
> com.apple.TimeMachine.2018-11-13-040721
> com.apple.TimeMachine.2018-11-13-051250
> 
> 
> So where would com.apple.TimeMachine.2018-11-12-060805  be located in
> the boot disk? "locate" doesn't find it.

Snapshots aren't located anywhere under / because that shows the current
state of the file system.

The snapshots are effectively parallel directories referring to the same
volume at earlier points in time.

The content of the snapshot would only be accessible (read only) via the
root file system if it was mounted, which is normally only done whie
Time Machine or some other utility is using the snapshot to do a backup.

-- 
David Empson
dempson@actrix.gen.nz

[toc] | [prev] | [next] | [standalone]

#119524

In message <r4yGD.202489$TK1.45120@fx19.iad> JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:
> On 2018-11-13 01:38, David Empson wrote:
>> tmutil listlocalsnapshots


> I get
> bike:~ $  tmutil listlocalsnapshots /
> com.apple.TimeMachine.2018-11-12-060805
> com.apple.TimeMachine.2018-11-12-070802
> com.apple.TimeMachine.2018-11-12-081408
> com.apple.TimeMachine.2018-11-12-091017
> com.apple.TimeMachine.2018-11-12-101438
> com.apple.TimeMachine.2018-11-12-111209
> com.apple.TimeMachine.2018-11-12-130716
> com.apple.TimeMachine.2018-11-12-140815
> com.apple.TimeMachine.2018-11-12-150756
> com.apple.TimeMachine.2018-11-12-161008
> com.apple.TimeMachine.2018-11-12-171211
> com.apple.TimeMachine.2018-11-12-181609
> com.apple.TimeMachine.2018-11-12-200722
> com.apple.TimeMachine.2018-11-12-211156
> com.apple.TimeMachine.2018-11-12-220722
> com.apple.TimeMachine.2018-11-12-231216
> com.apple.TimeMachine.2018-11-13-001553
> com.apple.TimeMachine.2018-11-13-011156
> com.apple.TimeMachine.2018-11-13-020721
> com.apple.TimeMachine.2018-11-13-040721
> com.apple.TimeMachine.2018-11-13-051250


> So where would com.apple.TimeMachine.2018-11-12-060805  be located in
> the boot disk? "locate" doesn't find it.

Locate doesn't fin many things, it's a 1990s era tool using 1970s file
access for Unix.

The snapshots are on the boot disk, as the command you issued showed
('/' is the boot disk).

-- 
Transvestite: A guy who likes to eat, drink and be Mary.

[toc] | [prev] | [next] | [standalone]

#119530

On 2018-11-13, Lewis <g.kreme@gmail.com.dontsendmecopies> wrote:
> In message <r4yGD.202489$TK1.45120@fx19.iad> JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:
>> On 2018-11-13 01:38, David Empson wrote:
>>> tmutil listlocalsnapshots
>
>> I get
>> bike:~ $  tmutil listlocalsnapshots /
>> com.apple.TimeMachine.2018-11-12-060805
>> com.apple.TimeMachine.2018-11-12-070802
>> com.apple.TimeMachine.2018-11-12-081408
>> com.apple.TimeMachine.2018-11-12-091017
>> com.apple.TimeMachine.2018-11-12-101438
>> com.apple.TimeMachine.2018-11-12-111209
>> com.apple.TimeMachine.2018-11-12-130716
>> com.apple.TimeMachine.2018-11-12-140815
>> com.apple.TimeMachine.2018-11-12-150756
>> com.apple.TimeMachine.2018-11-12-161008
>> com.apple.TimeMachine.2018-11-12-171211
>> com.apple.TimeMachine.2018-11-12-181609
>> com.apple.TimeMachine.2018-11-12-200722
>> com.apple.TimeMachine.2018-11-12-211156
>> com.apple.TimeMachine.2018-11-12-220722
>> com.apple.TimeMachine.2018-11-12-231216
>> com.apple.TimeMachine.2018-11-13-001553
>> com.apple.TimeMachine.2018-11-13-011156
>> com.apple.TimeMachine.2018-11-13-020721
>> com.apple.TimeMachine.2018-11-13-040721
>> com.apple.TimeMachine.2018-11-13-051250
>
>
>> So where would com.apple.TimeMachine.2018-11-12-060805  be located in
>> the boot disk? "locate" doesn't find it.
>
> Locate doesn't fin many things, it's a 1990s era tool using 1970s file
> access for Unix.

I'm surprised anyone still bothers with it these days.

Then again, this is coming from JF Mezei...

-- 
E-mail sent to this address may be devoured by my ravenous SPAM filter.
I often ignore posts from Google. Use a real news client instead.

JR

[toc] | [prev] | [next] | [standalone]

#119520

On 2018-11-13 01:38, David Empson wrote:

>> Or it it made is some hidden APFS partition/volume?
> 
> That wouldn't work. Snapshots must be on the volume they belong to.


I thought that snapshots could exist in the same APFS container as
separate volumes. if you need to create something inside the source
volume you are doing a snapshot of, then it no longer is a snapshot.

[toc] | [prev] | [next] | [standalone]

#119521

JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 2018-11-13 01:38, David Empson wrote:
> 
> >> Or it it made is some hidden APFS partition/volume?
> > 
> > That wouldn't work. Snapshots must be on the volume they belong to.
> 
> 
> I thought that snapshots could exist in the same APFS container as
> separate volumes. if you need to create something inside the source
> volume you are doing a snapshot of, then it no longer is a snapshot.

Snapshots are not separate volumes, but they behave somewhat like them,
and they can be mounted as read only volumes. A snapshot is tied to its
host volume.

Think of it like this:

- A volume has a directory to keep track of files inside it, in their
current state.

- A snapshot is another directory referencing the files on the same
volume, at an earlier point in time.

In effect, each volume has multiple directories: the active one (current
state) and any number of snapshots (earlier states).

In reality, the snapshots and current state are all part of a single
directory tree associated with the volume.

Go back and look at the APFS introduction in the WWDC presentations,
which explained the concept quite well, without going into
implementation details.

-- 
David Empson
dempson@actrix.gen.nz

[toc] | [prev] | [next] | [standalone]

#119531

On 2018-11-13 07:28, David Empson wrote:

> Snapshots are not separate volumes, but they behave somewhat like them,
> and they can be mounted as read only volumes. A snapshot is tied to its
> host volume.

By "Volume" you refer to a mountable file system that is one of possibly
multiple volumes inside an APFS container ? Or do you refer to an APFS
container itself (which exists as a GPT partition on the disk)


> In effect, each volume has multiple directories: the active one (current
> state) and any number of snapshots (earlier states).

Conceptually, do snapshots exist as sibblings ta same level as "/", but
because there is no command line means to access it, they remain hdden ?

Does Finder provide any access to them?

[toc] | [prev] | [next] | [standalone]

#119538

JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 2018-11-13 07:28, David Empson wrote:
> 
> > Snapshots are not separate volumes, but they behave somewhat like them,
> > and they can be mounted as read only volumes. A snapshot is tied to its
> > host volume.
> 
> By "Volume" you refer to a mountable file system that is one of possibly
> multiple volumes inside an APFS container ?

Yes.

If there are multiple volumes in an APFS container, each of those
volumes can have its own snapshots, which have nothing to do with other
volumes in the container (apart from sharing free space).

> > In effect, each volume has multiple directories: the active one (current
> > state) and any number of snapshots (earlier states).
> 
> Conceptually, do snapshots exist as sibblings ta same level as "/", but
> because there is no command line means to access it, they remain hdden ?

There is command line access to them, via tmutil and mount.

> Does Finder provide any access to them?

Not directly, but you can mount a snapshot (via the mount command line
tool), then access it as read-only volume in Finder.

Mounting them is a bit complicated as it needs an APFS-specific option,
but it didn't take me long to find the details via a Google search.

-- 
David Empson
dempson@actrix.gen.nz

[toc] | [prev] | [next] | [standalone]

#119543

On 2018-11-13 19:31, David Empson wrote:

>> Conceptually, do snapshots exist as sibblings ta same level as "/", but
>> because there is no command line means to access it, they remain hdden ?
> 
> There is command line access to them, via tmutil and mount.

Couldn't find info in neither high sierra man pages.

Google showed discussions on how this is possible but no solutions
offered. (and some apps will do this for you).

Is the snapshot akin to a hidden volume in the APFS container (sibbling
of main volume both childred of the APFS container and share space) or
does it exist inside the main volume (child of the main volume, and
grand-child of APFS container) ?


Other qiestion arisen from googling:

There is some mechanism that will trim the snapshots to keep some free
space in the APFS container.

Has Apple dopcumented what the free space target is below which it
starts trimming snapshots?

has it documented what triggers this check ? google mentioned that at
boot it may be trimmed. Would every backup of time machine do this too?


And finally, when in a restore side of Time Machine operations in the
GUI where it blanks all screens but uses only one, does this GUI
interface ever make use of the snapshots, or does it rely entirely on
the target backup drive's contents ?

[toc] | [prev] | [next] | [standalone]

#119545

JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 2018-11-13 19:31, David Empson wrote:
> 
> >> Conceptually, do snapshots exist as sibblings ta same level as "/", but
> >> because there is no command line means to access it, they remain hdden ?
> > 
> > There is command line access to them, via tmutil and mount.
> 
> Couldn't find info in neither high sierra man pages.

Then you didn't read the man pages properly. See everything mentioning
localsnapshot in the tmutil man page.

I'm assuming basic knowledge of how Unix systems in general mount other
volumes, and a basic level of comprehension of the man page for mount.

For mount, you need to read the man page for mount_apfs as that is
invoked by mount for mounting APFS volumes, and it documents how to
mount a snapshot.

> Google showed discussions on how this is possible but no solutions
> offered. (and some apps will do this for you).

Then you didn't look hard enough. I found the details quickly for
mounting a local snapshot, in a discussion off one of the first links I
tried. No Google search is needed for other manipulations of local
snapshots via tmutil because the man page covers it.

> Is the snapshot akin to a hidden volume in the APFS container (sibbling
> of main volume both childred of the APFS container and share space) or
> does it exist inside the main volume (child of the main volume, and
> grand-child of APFS container) ?

Why do you care? It makes no difference to how they get used.

Neither of your descriptions is correct, but the second one is closer.
Snapshots are not a "child" of anything. They are part of the volume's
directory. They can be thought of as siblings of the current state of
the directory, which can also be regarded as a snapshot (just one that
keeping getting updated, and isn't listed as a "local snapshot").

The way the system provides access to them makes them appear like they
are separate volumes (for mounting) but that is not how they are
actually implemented at the file system level.

They are certainly not separate volumes at the level of the APFS
container.

> Other qiestion arisen from googling:
> 
> There is some mechanism that will trim the snapshots to keep some free
> space in the APFS container.

Yes, which is implemented by calling tmutil thinlocalsnapshots.

> Has Apple dopcumented what the free space target is below which it
> starts trimming snapshots?

Not that I'm aware of.

The thinning mechanism supports an urgency parameter (again, see the man
page) which I expect the system uses by increasing the urgency if free
space is getting lower.

It is probably based on the previous logic used to manage the more
primitive local snapshots on HFS+ volumes.

> has it documented what triggers this check ? google mentioned that at
> boot it may be trimmed. Would every backup of time machine do this too?

Probably something is monitoring free space and triggering thinning if
it dips below certain thresholds, with extra checks at points such as
startup. Backups might trigger conditional thinning of older snapshots
if free space is low enough, there may also be a standard thinning
pattern like there is for external backup drives.

> And finally, when in a restore side of Time Machine operations in the
> GUI where it blanks all screens but uses only one, does this GUI
> interface ever make use of the snapshots, or does it rely entirely on
> the target backup drive's contents ?

It shows a mixture of both. If your external backup drive is connected,
you can restore from those backups. You can also restore from local
snapshots, and for APFS that is implemented by the Time Machine restore
inferface mounting the local snapshot to browse the files, copy, etc.,
unmounting it again when done.

-- 
David Empson
dempson@actrix.gen.nz

[toc] | [prev] | [next] | [standalone]

#119547

On 2018-11-14 18:04, David Empson wrote:

> Why do you care? It makes no difference to how they get used.

Because I try to understand how it is implemented. When problems arise,
understanding how the disk is used allows one to determine whether, in
this case, a snapshot is usable or not.



> Snapshots are not a "child" of anything. They are part of the volume's
> directory.

From mount_apfs: (as you suggested) :

 mount_apfs [-o options] -s snapshot pathname directory

-s snapshot   The name of the snapshot to mount.  In this usage pathname
is the
                   mounted root directory of the base volume containing
the snapshot.

The text is clear that a snapshoot is contained inside the volume. The
reference is to a mounted volume, not to a device.

compared to hdiutil which creates a virtual device that then can be
mounted as a normal disk). This creates a virtual volume that is not a
volume in the APFS container.

This matters because if, for some reason, you can't mount the main disk,
you can't access the snapshots contained within it.




> They can be thought of as siblings of the current state of
> the directory, 


"diskutil list"  does not show any of the snapshots as mountable volumes
within the APFS partition.  <boot_disk>, Preboot, Recovery and VM are
sibblings in the APFS container. Each is thoits own full fledge APFS
volume.

Since snapshots for DMA4 are contained within <boot disk>, they are not
sibblings to the others listed but rather children of DMA4.

Assuming EFI were able to parse an APFS container to find bootable
volumes, it would not find snapshots.



> The way the system provides access to them makes them appear like they
> are separate volumes (for mounting) but that is not how they are
> actually implemented at the file system level.

The way the system makes mounting of a .DMG makes it look like they are
separate volumes with their own device name (virtualized by hdiutil).
Yet the device maps to a file inside a mounted volume.

It appears snapshots are not files inside a mounted volume, but perhaps
data structures in the volume's metadate which allows a special driver
to create a vistualized mountable volume form that metadata which then
allows the apfs software to access that volume.


> It shows a mixture of both. If your external backup drive is connected,
> you can restore from those backups. You can also restore from local
> snapshots, and for APFS that is implemented by the Time Machine restore
> inferface mounting the local snapshot to browse the files, copy, etc.,
> unmounting it again when done.

Thankls for that info. I assume that if the designated target for time
machine (extremnal drive) is present, the snapshots are not really used,
but will automatically become visible if the backup drive is not accessible?

[toc] | [prev] | [next] | [standalone]

#119549

JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 2018-11-14 18:04, David Empson wrote:
> 
> > Snapshots are not a "child" of anything. They are part of the volume's
> > directory.
> 
> From mount_apfs: (as you suggested) :
> 
>  mount_apfs [-o options] -s snapshot pathname directory
> 
> -s snapshot   The name of the snapshot to mount.  In this usage pathname
> is the
>                    mounted root directory of the base volume containing
> the snapshot.
> 
> The text is clear that a snapshoot is contained inside the volume. The
> reference is to a mounted volume, not to a device.

A snapshot is part of the directory of a volume, so it is "inside" the
volume from that perspective. So is the current state of the directory.

> compared to hdiutil which creates a virtual device that then can be
> mounted as a normal disk). This creates a virtual volume that is not a
> volume in the APFS container.
> 
> This matters because if, for some reason, you can't mount the main disk,
> you can't access the snapshots contained within it.

Of course not. The volume needs to be able to be mounted to access
anything inside it.

Third party utility developers (e.g. Alsoft for DiskWarrior) are in the
process of implementing full support for APFS volumes, now that they
have the necessary informaton from Apple.

> > They can be thought of as siblings of the current state of
> > the directory, 
> 
> 
> "diskutil list"  does not show any of the snapshots as mountable volumes
> within the APFS partition.

diskutil knows nothing about snapshots because as I already said several
times, they are not separate volumes at the level of the container.

> <boot_disk>, Preboot, Recovery and VM are sibblings in the APFS container.
> Each is thoits own full fledge APFS volume.

None of that has anything to do with snapshots, because snapshots are
not separate volumes.

Snapshots are part of the directory of a volume.

> Since snapshots for DMA4 are contained within <boot disk>, they are not
> sibblings to the others listed but rather children of DMA4.

I never said they were siblings of other volumes.

Is "DMA4" supposed to mean something? I assume that is the name of your
drive.

> Assuming EFI were able to parse an APFS container to find bootable
> volumes, it would not find snapshots.

That probably depends on the preboot and recovery volumes and what they
are configured to access. It would not be normal to boot from a snapshot
so by default they would be ignored, but in principle it could be done
as long as the system can handle the constraint of booting a full system
from a read-only volume. It would just require the boot information to
point to a different snapshot in the volume's directory.

The recovery volume can also access snapshots on the main volume, via
the Time Machine full disk restore feature. That can be used to restore
the entire main volume to an earlier snapshot, including major things
like undoing an OS upgrade.

> > The way the system provides access to them makes them appear like they
> > are separate volumes (for mounting) but that is not how they are
> > actually implemented at the file system level.
> 
> The way the system makes mounting of a .DMG makes it look like they are
> separate volumes with their own device name (virtualized by hdiutil).
> Yet the device maps to a file inside a mounted volume.

Disk images involve a double layer of drivers: file system -> virtual
block driver -> file system -> real block driver.

Mounting of snapshots is done at the same level as mounting of volumes,
within the APFS file system driver. It just starts at a different
snapshot rather than the current state of the volume (which as I said is
effectively also a snapshot, without the restriction of being read
only).

> It appears snapshots are not files inside a mounted volume

Which I already said.

> but perhaps data structures in the volume's metadate which allows a
> special driver to create a vistualized mountable volume form that metadata
> which then allows the apfs software to access that volume.

They are a part of the volume directory structure, addressable (by the
APFS file system driver) separately from the root directory of the
current state of the volume.

> > It shows a mixture of both. If your external backup drive is connected,
> > you can restore from those backups. You can also restore from local
> > snapshots, and for APFS that is implemented by the Time Machine restore
> > inferface mounting the local snapshot to browse the files, copy, etc.,
> > unmounting it again when done.
> 
> Thankls for that info. I assume that if the designated target for time
> machine (extremnal drive) is present, the snapshots are not really used,
> but will automatically become visible if the backup drive is not accessible?

Based on what the Time Machine Restore interface is showing for me in
Mojave (and I think High Sierra was the same), it shows all known
backups on all configured backup drives, plus all local snapshots which
still remain.

There is no way to tell whether a specific entry in the timeline is an
external backup or local snapshot, except that if one of the external
backup drives is not available at that point, they are greyed out in the
timeline (and in the stack of window title bars).

In cases where the local snapshot still exists, I typically see a local
snapshot at the start time of the backup, plus an external backup at the
end time of the backup. For quick backups there may be two entries in
the same minute.

If you happen to pick a local snapshot, it uses the local snapshot to
restore files, including mounting it as I described previously.

If you happen to pick an external backup, it uses the external backup to
restore files and ignores the local snapshots.

-- 
David Empson
dempson@actrix.gen.nz

[toc] | [prev] | [next] | [standalone]

#119556

Thanks for your explanation.

Does APFS use a totally different architecture (catalogue file) from HFS+ ?

Are there actual directory files? or still one central database of files?

I ask in a context of how much must be replicated when snapshot is
taken, or whether it is *all* on a copy-on-wite principle where once
something changes in the real disk (such as renaming a folder), the
system gets a new block to store that changed information whereas the
snapshot keeps pointing to old block, even for the disk metadata.

[toc] | [prev] | [next] | [standalone]

#119571

On 2018-11-15 17:39:55 +0000, JF Mezei said:

> Does APFS use a totally different architecture (catalogue file) from HFS+ ?
> Are there actual directory files? or still one central database of files?

Since you want to understand, here are docs and tools:

http://newosxbook.com/files/APFS.pdf

Useful links at the end of that PDF, including to the Apple APFS docs.



-- 
Pure Personal Opinion | HoffmanLabs LLC 

[toc] | [prev] | [next] | [standalone]

#119575

On 2018-11-16 13:45, Stephen Hoffman wrote:

> http://newosxbook.com/files/APFS.pdf

Thanks. That doc didn't really explain, it sort of documented some of
the field found in some "include" files.

But the link to the Apple reference on APFS is very very valuable.

"For example, a drive can be configured with two bootable volumes"

That is new to me.  Was APFS earlier on only capable of 1?


BTW, when APFS says it uses 4096 byte blocks,  does the disk interpret a
request for block 2 as reading 4096 bytes from block 2,  or does the
disk still work in 512 blocks and the OS translate block into 512byte
blocks and request the disk read 8 blocks starting at block 9?

apfs_num_symlinks
The number of symbolic links in this volume.
uint64_t apfs_num_symlinks

Out of curiosity, why would the volume find it useful to keep track of
the number of symbolic links?


I had been laughed at by the regulars here on one of the problems with
APFS on spinning drives being fragmenntation, being told it doesn't
exist etc etc.

##
Warning
To avoid data corruption, this flag must not be set.
This flag enabled a prerelease version of the defragmentation system in
macOS 10.13 versions. Itʼs ignored by macOS 10.13.6 and later.
##

There was some major work done between early APFS and current one to
support spinning drives and that includes rewrite of defragmentation.
This points to this having been one of the problems that caused bad
performance on HDD in early APFS.



"The container layer organizes file-system layer information and stores
higher level information, such as volume metadata, snapshots of the
volume, and encryption state"

This is interesting because later on, the VOLUME header contains pointer
to a B-Tree list of snapshots.  So there is no clear answer on whether
snapshots exist as entities in the container or in the volume. What is
clear is that the Volume has a list of pointers to snapshots of itself.

(And the "man" text which stipilates that a volume must be mounted to
access its snapshots would point to the utilities using the volume
header blocks as the pointer to the list of snapshots.




What I have not yet been able to confirm is that APFS uses the same
"catalogue file" concept as HFS, namely a huge B-=tree structure that
contains records of various data types (directory, file etc).



The Apple document, like the other one, is big on defining fields, but
doesn't explain the general structure other than the concept of a
container which contains one or more volumes. Had been expecting such
information at WWDC2018 but there were no sessions like that.

That document also does not provide hints on how it manages
defragmentation and more importantly fusion drives. (defragmentation has
challenges when you have snapshots, and when you update the "live" file,
it becomes fragmented and the system wants to move blocks around to make
the updated file continuoius again (since in-situ writes not allowed)

[toc] | [prev] | [next] | [standalone]

#119577

JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 2018-11-16 13:45, Stephen Hoffman wrote:
> 
> > http://newosxbook.com/files/APFS.pdf
> 
> Thanks. That doc didn't really explain, it sort of documented some of
> the field found in some "include" files.
> 
> But the link to the Apple reference on APFS is very very valuable.

Thanks from me also, Stephen. I had already found and was trying to wade
through the Apple technical document, but it is not well organised for
getting a general overview.

The newosxbook presentation helped, and browsing a test disk image with
fsleuth clarified a few more details.

> "For example, a drive can be configured with two bootable volumes"
> 
> That is new to me.  Was APFS earlier on only capable of 1?

That may have been the case in APFS version 1 (developer preview on
Sierra), which Apple hasn't documented in detail. I haven't tried
multi-volume boot scenerios yet, except in separate
partitions/containers.

> BTW, when APFS says it uses 4096 byte blocks,  does the disk interpret a
> request for block 2 as reading 4096 bytes from block 2,  or does the
> disk still work in 512 blocks and the OS translate block into 512byte
> blocks and request the disk read 8 blocks starting at block 9?

That depends on the drive. Some use 512 byte blocks, some use 4096 byte
blocks, some can support either addressing scheme. The drive itself or
the device driver takes care of the mapping details.

> apfs_num_symlinks
> The number of symbolic links in this volume.
> uint64_t apfs_num_symlinks
> 
> Out of curiosity, why would the volume find it useful to keep track of
> the number of symbolic links?

For error checking. There are corresponding counters for number of
files, directories, other file objects and snapshots. When fsck checks
the file system it can confirm it found the right number of everything.

> "The container layer organizes file-system layer information and stores
> higher level information, such as volume metadata, snapshots of the
> volume, and encryption state"
> 
> This is interesting because later on, the VOLUME header contains pointer
> to a B-Tree list of snapshots.

I think the quoted sentence is in error about snapshots. I can't see any
other evidence of snapshot information being outside the scope of a
volume. Snapshot metadata is referenced from the volume superblock and
appears to contain everything needed to manage the snapshot (including
the transaction ID at which the snapshot was created).

> So there is no clear answer on whether snapshots exist as entities in the
> container or in the volume. What is clear is that the Volume has a list of
> pointers to snapshots of itself.

That is what "tmutil listlocalsnapshots /" shows, though that list may
be filtered to only list snapshots created by Time Machine and related
tools, not all current snapshots created directly through the API.

For example, Carbon Copy Cloner and SuperDuper can create a snpashot to
do backups, which is done directly through the API. They delete that
snapshot once it is no longer needed.

Creating snapshots requires an entitlement granted by Apple, so only
software Apple deems worthy can use the API. tmutil can create
snapshots, but only if Time Machine is configured (and the volume is
included in the backup).

> What I have not yet been able to confirm is that APFS uses the same
> "catalogue file" concept as HFS, namely a huge B-=tree structure that
> contains records of various data types (directory, file etc).

Yes. The volume superblock has an apfs_root_tree_oid field which points
(indirectly) to the root of a B-tree which contains the entire
filesystem of the volume.

Its internal structure and indexing arangement is very different to
HFS+. Changes to the filesystem only require copy-on-write replacement
of modified blocks of the B-tree.

For those aware of the horrible hack HFS+ had to use to implement hard
links, a major point is that the APFS file system tree has the inode
number as its primary index, and inode records contain reference counts
for use by hard links, which is the same way it is implemented on
standard UNIX file systems.

Unlike HFS+, there is no attempt to internally group file records by
their parent directory. Iterating through files in a directory requires
reading adjacent directory entry records to get the filename and inode
number, then looking up each inode number in the filesystem tree to get
the rest of the details about the file.

Typically the filesystem tree would be cached in memory so this is a
series of memory reads, not disk reads.

> The Apple document, like the other one, is big on defining fields, but
> doesn't explain the general structure other than the concept of a
> container which contains one or more volumes. Had been expecting such
> information at WWDC2018 but there were no sessions like that.
> 
> That document also does not provide hints on how it manages
> defragmentation and more importantly fusion drives.

The Apple document covers features implemented in macOS 10.13 and says
at the beginning it doesn't cover encryption or Fusion drives (and by
extension the defragmentation scheme). It also looks like some parts are
not written yet as they say "No overview available".

There is some information about Fusion drives in there, e.g. showing the
ability to tag files to remain on the SSD, mention of using the SSD as a
cache for the hard drive, and a special mode for small capacity SSDs
(presumably the 64 GB one used in recent mdoels with smaller capacity
hard drives).

We may see an updated version of the document in future.

> (defragmentation has challenges when you have snapshots, and when you
> update the "live" file, it becomes fragmented and the system wants to move
> blocks around to make the updated file continuoius again (since in-situ
> writes not allowed)

I haven't looked closely at the details of extents yet but in principle
the defragmenter just needs to move file extents to be in adjacent
physical blocks, considering the current state of the file system
(latest transaction ID for each object ID for the extents that make up a
file). That would suffice for minimising hard drive head movement, at
least for the current state of the file (or one instance of the file if
it is has been cloned and then partly modified).

Going further and merging adjacent extents is probably more trouble than
it is worth. The only gains would be reducing internal fragmentation and
freeing entries for keeping track of the extents, which in most cases
would be a tiny space saving relative to the drive capacity. This
wouldn't work if the extents were referenced by multiple clones or
snapshots with differences in the extent list, so it is easier to not
bother trying.

-- 
David Empson
dempson@actrix.gen.nz

[toc] | [prev] | [next] | [standalone]

#119579

On 2018-11-18 00:37, David Empson wrote:

> That may have been the case in APFS version 1 (developer preview on
> Sierra), which Apple hasn't documented in detail. I haven't tried
> multi-volume boot scenerios yet, except in separate
> partitions/containers.

When I watched the APFS presentation in 2017, it mentiond that APFS
basically placed the boot information (block location, number of blocks
to load etc) in block 0 of the GPT partition where the EFI firmware
would look to see if the GPT partition was bootable.

So from an EFI point of view, it could only see one bootable volume in
the GPT partition that containes the APFS container which may contain
multiple bootable APFS volumes)

Considering the recovery partition is also bootable and exists in the
same APFS container, this would likekly require that a neutral "pre
boot" environment is what EFI actually loads and that pre-boot then
decides which APFS volume inside the APFS container is to be loaded.
(the pre-boot stuff is what the EFI partition was actually meant to do,
as well as be the one which understands how to parse the file system
containing the boot volume to load the primary bootstrap file.

And with encryption, a pre-boot middleware between EFO and actuao boot
would be required so that something can ask for password needed to
decrupt the boot drive before being able to load the primary bootstrap
for the OS.


But that would mean that at the EFI level, it cannot list all bootable
volumes, as it only sees bootable GPT partitions. (unless Apple adds
APFS understanding to the EFI firmware inside its computers)



>> "The container layer organizes file-system layer information and stores
>> higher level information, such as volume metadata, snapshots of the
>> volume, and encryption state"
>> 
>> This is interesting because later on, the VOLUME header contains pointer
>> to a B-Tree list of snapshots.
> 
> I think the quoted sentence is in error about snapshots. I can't see any
> other evidence of snapshot information being outside the scope of a
> volume.


Is it possible that space used by the snapshot (keeping certain blocks
"alive" when the source volume has released them) gets "charged" to the
snapshot at the APFS container level instead of being considered as part
of disk blocks used by the volume ? (which  would then not jive with
totaal blcoks used by files in that volume)



> That is what "tmutil listlocalsnapshots /" shows, though that list may
> be filtered to only list snapshots created by Time Machine and related
> tools, not all current snapshots created directly through the API.

list local snapshots gives names of snapshots in pretty output.
Consider a scenario where snapthots were indexed in the container for
all APFS volumes in that container.  Listlocalsnapshots / would simply
list those snamshots that belong to the current device that is the file
sytstem root. The output of the utility would be identical, even though
th structure deep down would be very different.



> Yes. The volume superblock has an apfs_root_tree_oid field which points
> (indirectly) to the root of a B-tree which contains the entire
> filesystem of the volume.

Yeah, I ended up making that conclusion, but I assume the document just
assumed everyone knew that the concept of a catalogue file from HFS was
maintained.


> Its internal structure and indexing arangement is very different to
> HFS+. Changes to the filesystem only require copy-on-write replacement
> of modified blocks of the B-tree.

In VMS for instance, the equivalemt of the catalogue file was INDEXF.SYS
(but only had file entries, not directory hiearchy information), and
each record had a fixed amount of space for extents, and when a file
became badly fragmented, an additional "continuation" record was created
to contain the additional extents.

Considering how APFS has gone "extent crazy" potentially resulting large
files with 1 extent per block, I am curious how they designed the
records in the big Btree catalogue to handle large number of extents.

I take it a snapshot has its own btree catalogue of the files that it
contains? or does it share use of the volume's catalogue B-tree ?

> For those aware of the horrible hack HFS+ had to use to implement hard
> links, a major point is that the APFS file system tree has the inode
> number as its primary index,

Yeah, it appears that they kept the same overall architecture for the
file system but added the fields needed to make it far more efficient.

> Unlike HFS+, there is no attempt to internally group file records by
> their parent directory. 

is the file AAAA.readme.txt stored anywhere in the b-tree (with primary
key being inode) or is some alphabetical order maintained in the b-tree?

In other words would
/etc/bind/myconfig/AAAA.readme    and
/System/Library/launchdaemons/documentation/AAAA.readme

be near each other in the B-tree or be very far apart ?


> I haven't looked closely at the details of extents yet but in principle
> the defragmenter just needs to move file extents to be in adjacent
> physical blocks, 

But when I update block 75 or a 200 block file, block 75 get written far
away and the file then becomes an extent for 0-74, one for 75 and one
for 76-200.

The problem is that a snapshop would retain single 0-200 extent with its
block 75 containing the old informaion.

When a defragmenter wants to recreate the live file to have one
conti8guious extent, it would move blocks 0-74 and 76-200 somewhere
else.  It would likely then have to update the snapshot to have blocks
0-74 poiting to the new location of the defragmented live file, block 74
pointing to the old location and blcosk 76-200 to the new location. (in
essence, the snapshot becomes fragmented in order to defragment the live
file. So when you have many snapshots, this can become inwieldy when you
often update random blocks in a file.

I was really expecting/hoping for an APFS update at this year WWDC to
discuss how they tackled that challenge.

[toc] | [prev] | [next] | [standalone]

#119581

JF Mezei <jfmezei.spamnot@vaxination.ca> wrote:

> On 2018-11-18 00:37, David Empson wrote:
> 
> > That may have been the case in APFS version 1 (developer preview on
> > Sierra), which Apple hasn't documented in detail. I haven't tried
> > multi-volume boot scenerios yet, except in separate
> > partitions/containers.
> 
> When I watched the APFS presentation in 2017, it mentiond that APFS
> basically placed the boot information (block location, number of blocks
> to load etc) in block 0 of the GPT partition where the EFI firmware
> would look to see if the GPT partition was bootable.

That information is only for locating and loading the APFS EFI jumpstart
code which is needed to boot from an APFS contaner.

> So from an EFI point of view, it could only see one bootable volume in
> the GPT partition that containes the APFS container which may contain
> multiple bootable APFS volumes)

EFI can't see any volumes, just the jumpstart bootloader (and the name
of the container).

Multiple bootable volumes in an APFS container is already a standard
feature: the recovery volume and the main volume are both bootable. The
EFI jumpstart driver switches to booting from the recovery volume if it
can't boot from the main volume, or if certain keys are held down.

In a more general case of multiple user-created bootable volumes (if
that is possible), it would just be a case of picking one of them as the
one which can currently be booted, via Startup Disk in System
Preferences (which updates the Preboot volume and other information as
required). The EFI Startup Manager (option key at startup) wouldn't be
able to pick between multiple bootable volumes inside the APFS
container.

> >> "The container layer organizes file-system layer information and stores
> >> higher level information, such as volume metadata, snapshots of the
> >> volume, and encryption state"
> >> 
> >> This is interesting because later on, the VOLUME header contains pointer
> >> to a B-Tree list of snapshots.
> > 
> > I think the quoted sentence is in error about snapshots. I can't see any
> > other evidence of snapshot information being outside the scope of a
> > volume.
> 
> Is it possible that space used by the snapshot (keeping certain blocks
> "alive" when the source volume has released them) gets "charged" to the
> snapshot at the APFS container level instead of being considered as part
> of disk blocks used by the volume ? (which  would then not jive with
> totaal blcoks used by files in that volume)

No. All information needed to track current and retained disk blocks is
available within the current and previous state of the objects within
the volume.

Everything is done through transactions. When an object (e.g. part of a
file) is modified, a new copy of that object and parts of everything
else affected (filesystem tree, object map and volume header) gets
written again using the same virtual object IDs as before but with a new
transaction ID.

There are multiple instances of each object as the volume changes (same
object ID, different transaction ID), so the reaper needs to work out
which older ones can be deleted. In the simplest case (no snapshots) for
an object with two or more instances, only the one with the latest
transaction ID needs to be retained.

Snapshots record a transaction ID, so they add another point in the
transaction history at which objects need to be retained. If the object
exists with multiple transaction IDs between two snapshots (or between
the last snapshot and the current state) then all but the newest one in
that range can be deleted.

> > That is what "tmutil listlocalsnapshots /" shows, though that list may
> > be filtered to only list snapshots created by Time Machine and related
> > tools, not all current snapshots created directly through the API.
> 
> list local snapshots gives names of snapshots in pretty output.
> Consider a scenario where snapthots were indexed in the container for
> all APFS volumes in that container. 

I won't consider that because it isn't how APFS implements snapshots.

> > Yes. The volume superblock has an apfs_root_tree_oid field which points
> > (indirectly) to the root of a B-tree which contains the entire
> > filesystem of the volume.
> 
> Yeah, I ended up making that conclusion, but I assume the document just
> assumed everyone knew that the concept of a catalogue file from HFS was
> maintained.

One of the reasons they went with a similar structure is that it makes
HFS+ to APFS conversion quite easy: an APFS filesystem tree is
constructed by copying data from the HFS+ catalog tree (and related
trees), without needing to modify or move file data.

> > Its internal structure and indexing arangement is very different to
> > HFS+. Changes to the filesystem only require copy-on-write replacement
> > of modified blocks of the B-tree.
> 
> In VMS for instance, the equivalemt of the catalogue file was INDEXF.SYS
> (but only had file entries, not directory hiearchy information), and
> each record had a fixed amount of space for extents, and when a file
> became badly fragmented, an additional "continuation" record was created
> to contain the additional extents.
> 
> Considering how APFS has gone "extent crazy" potentially resulting large
> files with 1 extent per block

What gives you that idea? If a file is created with a size known in
advance, APFS can create it as a single extent.

APFS is able to handle multiple extents per file, and may need to split
a large extent into multiple smaller extents, e.g. when part of a large
file is modiifed.

> I am curious how they designed the records in the big Btree catalogue to
> handle large number of extents.

Each file has an arbitrary number of records in the filesystem tree,
starting with an inode. It may include any number of extent records. All
records for the file are stored sequentially.

> I take it a snapshot has its own btree catalogue of the files that it
> contains?

No. A snapshot contains only its name, timestamp information and a
transaction ID.

Adding the snapshot will require adding it to the list of snapshots, so
a small portion of the volume is modified in the process, which will be
part of the same tranaction ID as the snapshot.

The content of the snapshot is identified by locating the volume
superblock, filesystem tree and file content objects which have the
appropriate object IDs and a transaction ID which is less than or equal
to the transaction ID of the snapshot.

> or does it share use of the volume's catalogue B-tree ?

Yes (as subsequent changes occur, the snapshot will keep referencing a
historic copy of the modified parts of the filesystem tree).

> > Unlike HFS+, there is no attempt to internally group file records by
> > their parent directory. 
> 
> is the file AAAA.readme.txt stored anywhere in the b-tree (with primary
> key being inode) or is some alphabetical order maintained in the b-tree?

When that file is created it will be allocated the next available inode
number for the volume. That is used to locate its inode record in the
filesystem tree.

The inode record does NOT contain the filename: the name is in the
appropriate directory entry record for its parent directory (separately
for each hard link to the file).

I didn't notifice any specification of ordering of directory entries
within directories but I haven't looked closely at that detail. There is
a hashing mechanism, which probably means they are sorted in hash order
rather than by filename: this allows quick location of a file by name,
but avoids needing to rewrite the entire directory for a single added or
removed file.

The difference I was commenting on above is where the inode records are
located.

HFS+: each directory contains an array of indoe records for files in
that directory, sorted in order of filename. The filename is inside the
inode record.

APFS: each directory contains directory entry recordsfor files in that
directory, which include the filename and inode number of the file. The
inode records for the files are stored elsewhere (indexed by their inode
number).

If you move a file to a different directory, HFS+ has to remove a
relatively large inode record from the old directory and insert it into
the new directory. APFS just needs to remove and insert a smaller
directory entry record.

> In other words would
> /etc/bind/myconfig/AAAA.readme    and
> /System/Library/launchdaemons/documentation/AAAA.readme
> 
> be near each other in the B-tree or be very far apart ?

Far apart, unless they happened to have inode numbers which are near
each other (e.g. if you happened to create those two files one after the
other).

HFS+ wouldn't have them anywhere near each other. It sorts by filename
within directories, not across the whole volume.

If you had /etc/bind/myconfig/AAAA and /etc/bind/myconfig/AAAB then
their inode records would be adjacent in the catalog tree for HFS+, but
could be widely separated for APFS (if those files were created at
different times), however APFS would have directory entry records near
each other (not necessarily adjacent) for those two files because they
are in the same parent directory.

> > I haven't looked closely at the details of extents yet but in principle
> > the defragmenter just needs to move file extents to be in adjacent
> > physical blocks, 

And as such I'm not going to comment on your speculation what happens
for defragmenting in a particular case involving a snapshot and modified
file.

-- 
David Empson
dempson@actrix.gen.nz

[toc] | [prev] | [next] | [standalone]

Page 3 of 5 — ← Prev page 1 2 [3] 4 5  Next page →

Back to top | Article view | comp.sys.mac.system

csiph-web