Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > alt.os.linux > #54222 > unrolled thread

freshly installed SW not found

Back to article view | Back to alt.os.linux

Contents

  freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-26 19:47 +0100
    Re: freshly installed SW not found "J.O. Aho" <user@example.net> - 2019-02-26 20:28 +0100
      Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-26 21:51 +0100
        Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-26 21:46 +0000
          Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 12:52 +0100
            Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-02-27 12:44 +0000
              Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-02-27 13:27 +0000
              Re: freshly installed SW not found Richard Kettlewell <invalid@invalid.invalid> - 2019-02-27 15:16 +0000
              Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 16:59 +0100
                Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-02-27 16:02 +0000
                  Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 20:46 +0100
                    Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-27 20:18 +0000
                      Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-28 12:47 +0100
                        Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-28 18:32 +0000
                          Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-03-01 01:14 +0100
                            Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-01 01:54 +0100
                        Re: freshly installed SW not found Jasen Betts <jasen@xnet.co.nz> - 2019-03-01 03:09 +0000
                    Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-02-27 20:21 +0000
                      Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-27 23:02 +0000
                        Re: freshly installed SW not found Richard Kettlewell <invalid@invalid.invalid> - 2019-02-27 23:24 +0000
                        Re: freshly installed SW not found Jasen Betts <jasen@xnet.co.nz> - 2019-02-28 04:46 +0000
                    Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-02-28 10:59 +0000
                      Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-28 18:51 +0100
                        Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-28 18:24 +0000
                        Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-03-01 13:49 +0000
                          Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-01 18:22 +0100
                            Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-03-01 19:06 +0000
                          Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-03-03 01:08 +0100
                            Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-03-03 00:14 +0000
                              Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-03-03 13:00 +0000
                            Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-03-03 12:46 +0000
                      Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-28 18:15 +0000
                Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-27 16:38 +0000
                  Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 20:51 +0100
                Re: freshly installed SW not found Jasen Betts <jasen@xnet.co.nz> - 2019-02-28 04:39 +0000
          Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-02-27 14:30 +0100
        Re: freshly installed SW not found "J.O. Aho" <user@example.net> - 2019-02-27 07:05 +0100
        Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-02-27 14:29 +0100
          Re: freshly installed SW not found Paul <nospam@needed.invalid> - 2019-02-27 08:50 -0500
            Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-02-28 00:05 +0100
          Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 16:56 +0100
          Re: freshly installed SW not found Chris <ithinkiam@gmail.com> - 2019-03-01 14:58 +0000
        Re: freshly installed SW not found Chris <ithinkiam@gmail.com> - 2019-03-01 15:04 +0000
          Re: freshly installed SW not found "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-03-01 10:50 -0500
            Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-03-01 18:40 +0000
              Re: freshly installed SW not found "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-03-01 14:40 -0500
                Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-03-01 23:38 +0000
                  Re: freshly installed SW not found "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-03-01 19:09 -0500
                    Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-03-02 03:47 +0000
                      Re: freshly installed SW not found "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-03-01 22:55 -0500
                        Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-03-02 07:08 +0000
                          Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-02 11:27 +0100
                      Re: freshly installed SW not found Paul <nospam@needed.invalid> - 2019-03-02 04:10 -0500
                        Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-03-02 16:29 +0000
                          Re: freshly installed SW not found Paul <nospam@needed.invalid> - 2019-03-02 11:54 -0500
                      Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-02 11:24 +0100
                  Re: freshly installed SW not found Jasen Betts <jasen@xnet.co.nz> - 2019-03-02 00:28 +0000
                    Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-02 11:31 +0100
                      Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-03-02 11:31 +0000
                        Re: freshly installed SW not found Melzzzzz <Melzzzzz@zzzzz.com> - 2019-03-02 11:34 +0000
                          Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-03-02 11:48 +0000
                            Re: freshly installed SW not found Melzzzzz <Melzzzzz@zzzzz.com> - 2019-03-02 12:15 +0000
                              Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-02 23:15 +0100
                                Re: freshly installed SW not found Chris Elvidge <chris@mshome.net> - 2019-03-03 00:33 +0000
                                Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-03-03 01:38 +0100
                                  Compressed filesystems [Was: Re: freshly installed SW not found] "Carlos E.R." <robin_listas@es.invalid> - 2019-03-03 13:11 +0100
                                    Re: Compressed filesystems [Was: Re: freshly installed SW not found] Soviet_Mario <SovietMario@CCCP.MIR> - 2019-03-03 19:49 +0100
                                      Re: Compressed filesystems [Was: Re: freshly installed SW not found] "Carlos E.R." <robin_listas@es.invalid> - 2019-03-03 21:17 +0100
                                  Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-03-03 12:49 +0000
                                Re: freshly installed SW not found Melzzzzz <Melzzzzz@zzzzz.com> - 2019-03-03 12:35 +0000
              Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-02 11:22 +0100
    Re: freshly installed SW not found Wildman <best_lay@yahoo.com> - 2019-02-26 13:31 -0600
      Re: freshly installed SW not found Wildman <best_lay@yahoo.com> - 2019-02-26 15:42 -0600
        Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-26 21:51 +0000
    Re: freshly installed SW not found "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-02-26 14:36 -0500
      Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-26 21:58 +0100
        Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-26 21:49 +0000
          Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 12:54 +0100
    Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-26 19:51 +0000

Page 2 of 4 — ← Prev page 1 [2] 3 4  Next page →

#54258

On 2019-02-27, William Unruh <unruh@invalid.ca> wrote:
> On 2019-02-27, Aragorn <thorongil@telenet.be> wrote:

>  It has never been entirely clear to me if the inode saves a reference
>  to the files pointing to it. Ie, if you know the inode, can you
>  determine all of the files that are hard linked to it, without haveing
>  to scan the whole directory structure to fine those files that point to
>  it.

You have to scan.

>  For example, rsync can preserve hard links. How does it know that
>  file B is a hardlink to a file already read as file A, if it does?

It remembers the inode numbers of the files it has found. if they
match (and are on the same filesystem) then it knows there's a hard
link.

(it probably only remembers inode numbers for files having more than one
in the link count. check the source for details)


-- 
  When I tried casting out nines I made a hash of it.

[toc] | [prev] | [next] | [standalone]

#54259

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Soviet_Mario wrote:
> On 27/02/19 17:02, Dan Purgert wrote:
>> Soviet_Mario wrote:
>>> On 27/02/19 13:44, Aragorn wrote:
>>>> [...]
>>>> rsnapshot uses hard-linking for files that haven't changed since
>>>> the last backup.  Hard links don't use up any inodes.
>>>>
>>>
>>> mmm, then I must have chosen the wrong terms.
>>> But some other kind of disk space must be used to add a link
>> 
>> Nope.
>
> so the link information is stored where ?
> I'm not saying the content need to be stored, but the link 
> to it for sure : in as many places as there are the links

On the filesystem/directory metadata ("in" the directory, but it doesn't
take space.  Or well, worst case, it spills into the next sector of
metadata, so burns however much space your sector size is -- generally
somewhere between 512 and 4096 bytes).

>
> This does not duplicate the info of Inode (Aragorns taught), 
> but then the link is a reference to be dereferenced. So when 
> the file manager scans the disk, re-read the same inode many 
> times or at best looks-up some hash table to find whether or 
> not the info is just present.

You're crossing hard-links (file A and file B are exactly the same
inode, there is no reference / dereference), and soft-links (file A
[inode 1234] points to fileB [inode 5678])

>>> to somewhere. And this huge list of items (be them inode or
>>> else) really slows down directory listing and scan, for
>>> example when I perform searches or get properties (counts,
>>> stats etc)
>> 
>> Which'll be as equally slow, if not slower, when you try doing it on a
>> tarball.
>
> If one has just a huge one, for sure. That's why I wanted 
> tarbal:folder=1:1

Er, which means that you have to untar the whole thing, at least into
memory.  Which, depending on your hardware will result in no gains, or a
net slowdown.

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEBcqaUD8uEzVNxUrujhHd8xJ5ooEFAlx3vyYACgkQjhHd8xJ5
ooFN6gf+JiyCsXgASnN/6RNtXF/G8Lb9Nd0hP8E7qvBA6XUbSfvph9XUd8bb8K2T
x1+f1vy6PfouRz+IYbPKneNyCnNasrqnFCTL+9FmeNBHY3L2sUSzdl6+iJg57AYX
r/h6fonI9mCA+POD4Z/Ed6lXpXsW/isQVVDpq8irMfDPSUsG5jIFhb5fb6WGpETJ
MFD6rN7Cu9FjSH39psWD5l9pnPZsaSy8Vv6yT1TRRrMan/UDw4zoV3K4ynNDi4Io
oyE+740dfijcXwp9eqya/bRb7srWw97mTfcNSe+3MVV+xXbqD6jovkjEdDax4QVX
Ce9eDRwvqIDP2gTTGmKak6reRmcdzw==
=GpRo
-----END PGP SIGNATURE-----

-- 
|_|O|_| 
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

[toc] | [prev] | [next] | [standalone]

#54261

On 28/02/19 11:59, Dan Purgert wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> Soviet_Mario wrote:
>> On 27/02/19 17:02, Dan Purgert wrote:
>>> Soviet_Mario wrote:
>>>> On 27/02/19 13:44, Aragorn wrote:
>>>>> [...]
>>>>> rsnapshot uses hard-linking for files that haven't changed since
>>>>> the last backup.  Hard links don't use up any inodes.
>>>>>
>>>>
>>>> mmm, then I must have chosen the wrong terms.
>>>> But some other kind of disk space must be used to add a link
>>>
>>> Nope.
>>
>> so the link information is stored where ?
>> I'm not saying the content need to be stored, but the link
>> to it for sure : in as many places as there are the links
> 
> On the filesystem/directory metadata ("in" the directory, but it doesn't
> take space. 

i reckon it does, even if it takes "reserved" space. And 
anyway it takes time to be readed

> Or well, worst case, it spills into the next sector of
> metadata, so burns however much space your sector size is -- generally
> somewhere between 512 and 4096 bytes).
> 
>>
>> This does not duplicate the info of Inode (Aragorns taught),
>> but then the link is a reference to be dereferenced. So when
>> the file manager scans the disk, re-read the same inode many
>> times or at best looks-up some hash table to find whether or
>> not the info is just present.
> 
> You're crossing hard-links (file A and file B are exactly the same
> inode, 

I don't miss this point, I was saying a different thing : 
this same inode is listed under as many directories as the 
hardlinks

> there is no reference / dereference), and soft-links (file A
> [inode 1234] points to fileB [inode 5678])

yes to be able to cross FS boundaries

> 
>>>> to somewhere. And this huge list of items (be them inode or
>>>> else) really slows down directory listing and scan, for
>>>> example when I perform searches or get properties (counts,
>>>> stats etc)
>>>
>>> Which'll be as equally slow, if not slower, when you try doing it on a
>>> tarball.
>>
>> If one has just a huge one, for sure. That's why I wanted
>> tarbal:folder=1:1
> 
> Er, which means that you have to untar the whole thing, at least into
> memory.  Which, depending on your hardware will result in no gains, or a
> net slowdown.

lets put it from start.

Long time ago in a distant galaxy, there was a file system 
structure very deeply nested and complex, but much static in 
its general shape and very well known and familiar to its owner

The contents changed here and there.
This wierd owner desired two different and contradictory 
goals : to have phisical copies of the updates versions 
(distinct files) while at the same time sharing one single 
copy of the folder structure (or a "super-set" when some 
change to it occurred), as in crawling in the tree he 
seldomly required search.
to be continued...


> 
> -----BEGIN PGP SIGNATURE-----
> 
> iQEzBAEBCAAdFiEEBcqaUD8uEzVNxUrujhHd8xJ5ooEFAlx3vyYACgkQjhHd8xJ5
> ooFN6gf+JiyCsXgASnN/6RNtXF/G8Lb9Nd0hP8E7qvBA6XUbSfvph9XUd8bb8K2T
> x1+f1vy6PfouRz+IYbPKneNyCnNasrqnFCTL+9FmeNBHY3L2sUSzdl6+iJg57AYX
> r/h6fonI9mCA+POD4Z/Ed6lXpXsW/isQVVDpq8irMfDPSUsG5jIFhb5fb6WGpETJ
> MFD6rN7Cu9FjSH39psWD5l9pnPZsaSy8Vv6yT1TRRrMan/UDw4zoV3K4ynNDi4Io
> oyE+740dfijcXwp9eqya/bRb7srWw97mTfcNSe+3MVV+xXbqD6jovkjEdDax4QVX
> Ce9eDRwvqIDP2gTTGmKak6reRmcdzw==
> =GpRo
> -----END PGP SIGNATURE-----
> 


-- 
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
Soviet_Mario - (aka Gatto_Vizzato)

[toc] | [prev] | [next] | [standalone]

#54263

On 2019-02-28, Soviet_Mario <SovietMario@CCCP.MIR> wrote:
> On 28/02/19 11:59, Dan Purgert wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>> 
>> Soviet_Mario wrote:
>>> On 27/02/19 17:02, Dan Purgert wrote:
>>>> Soviet_Mario wrote:
>>>>> On 27/02/19 13:44, Aragorn wrote:
>>>>>> [...]
>>>>>> rsnapshot uses hard-linking for files that haven't changed since
>>>>>> the last backup.  Hard links don't use up any inodes.
>>>>>>
>>>>>
>>>>> mmm, then I must have chosen the wrong terms.
>>>>> But some other kind of disk space must be used to add a link
>>>>
>>>> Nope.
>>>
>>> so the link information is stored where ?
>>> I'm not saying the content need to be stored, but the link
>>> to it for sure : in as many places as there are the links
>> 
>> On the filesystem/directory metadata ("in" the directory, but it doesn't
>> take space. 
>
> i reckon it does, even if it takes "reserved" space. And 
> anyway it takes time to be readed

ANY file the metadata has to be read to find out where on the disk to go
to get the data. That on the Linux systems is the inode. That inode HAS
to be read to figure out where the data is. It just so happens that
there may be many files that point to the same inode. The files do not
know of this. The inode itself has a few bytes which lists how many
files point to it, not which files, just the number. Thus you have
thousands or millions of files pointing to the inode and it will not
take up any room on the inode, nor any room on the file listing. You
will of course have those millions of directory entries, but so what.
You are the one that wanted them. 

>
>> Or well, worst case, it spills into the next sector of
>> metadata, so burns however much space your sector size is -- generally
>> somewhere between 512 and 4096 bytes).
>> 
>>>
>>> This does not duplicate the info of Inode (Aragorns taught),
>>> but then the link is a reference to be dereferenced. So when
>>> the file manager scans the disk, re-read the same inode many
>>> times or at best looks-up some hash table to find whether or
>>> not the info is just present.
>> 
>> You're crossing hard-links (file A and file B are exactly the same
>> inode, 
>
> I don't miss this point, I was saying a different thing : 
> this same inode is listed under as many directories as the 
> hardlinks

But every entry in the directory would have to have an inode pointer for
each of the files. 

>
>> there is no reference / dereference), and soft-links (file A
>> [inode 1234] points to fileB [inode 5678])
>
> yes to be able to cross FS boundaries

NO. hard links CANNOT cross FS boundaries. 

>
>> 
>>>>> to somewhere. And this huge list of items (be them inode or
>>>>> else) really slows down directory listing and scan, for
>>>>> example when I perform searches or get properties (counts,
>>>>> stats etc)
>>>>
>>>> Which'll be as equally slow, if not slower, when you try doing it on a
>>>> tarball.
>>>
>>> If one has just a huge one, for sure. That's why I wanted
>>> tarbal:folder=1:1
>> 
>> Er, which means that you have to untar the whole thing, at least into
>> memory.  Which, depending on your hardware will result in no gains, or a
>> net slowdown.
>
> lets put it from start.
>
> Long time ago in a distant galaxy, there was a file system 
> structure very deeply nested and complex, but much static in 
> its general shape and very well known and familiar to its owner
>
> The contents changed here and there.
> This wierd owner desired two different and contradictory 
> goals : to have phisical copies of the updates versions 
> (distinct files) while at the same time sharing one single 
> copy of the folder structure (or a "super-set" when some 
> change to it occurred), as in crawling in the tree he 
> seldomly required search.
> to be continued...

What in the world does "a superset" ,mean. On my system some files
change hourly. Do you really mean some weird tree structure which you
have to wind you way down everytime you want a file?

Anyway why don't you write it. See if anyone else has the same wants as
you?

>
>
>> 
>> -----BEGIN PGP SIGNATURE-----
>> 
>> iQEzBAEBCAAdFiEEBcqaUD8uEzVNxUrujhHd8xJ5ooEFAlx3vyYACgkQjhHd8xJ5
>> ooFN6gf+JiyCsXgASnN/6RNtXF/G8Lb9Nd0hP8E7qvBA6XUbSfvph9XUd8bb8K2T
>> x1+f1vy6PfouRz+IYbPKneNyCnNasrqnFCTL+9FmeNBHY3L2sUSzdl6+iJg57AYX
>> r/h6fonI9mCA+POD4Z/Ed6lXpXsW/isQVVDpq8irMfDPSUsG5jIFhb5fb6WGpETJ
>> MFD6rN7Cu9FjSH39psWD5l9pnPZsaSy8Vv6yT1TRRrMan/UDw4zoV3K4ynNDi4Io
>> oyE+740dfijcXwp9eqya/bRb7srWw97mTfcNSe+3MVV+xXbqD6jovkjEdDax4QVX
>> Ce9eDRwvqIDP2gTTGmKak6reRmcdzw==
>> =GpRo
>> -----END PGP SIGNATURE-----
>> 
>
>

[toc] | [prev] | [next] | [standalone]

#54268

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Soviet_Mario wrote:
> On 28/02/19 11:59, Dan Purgert wrote:
>> Soviet_Mario wrote:
>>> On 27/02/19 17:02, Dan Purgert wrote:
>>>> Soviet_Mario wrote:
>>>>> On 27/02/19 13:44, Aragorn wrote:
>>>>>> [...]
>>>>>> rsnapshot uses hard-linking for files that haven't changed since
>>>>>> the last backup.  Hard links don't use up any inodes.
>>>>>>
>>>>>
>>>>> mmm, then I must have chosen the wrong terms.
>>>>> But some other kind of disk space must be used to add a link
>>>>
>>>> Nope.
>>>
>>> so the link information is stored where ?
>>> I'm not saying the content need to be stored, but the link
>>> to it for sure : in as many places as there are the links
>> 
>> On the filesystem/directory metadata ("in" the directory, but it doesn't
>> take space. 
>
> i reckon it does, even if it takes "reserved" space. And 
> anyway it takes time to be readed

You reckon wrong.  Upon creation, of the filesystem, the space is
already used (for the inode itself).  After that, you just update the
inode to say how many "files(tm)" are hard-linked to it.

It doesn't matter if there's only one file linked to the inode, or a
thousand - you still have to read that inode to know where the data
actually is.

It's faster to read a thousand "files" linked to the same inode, than
1000 separate inodes, as you're only reading metadata (contained in
tightly grouped tables) rather than having to jump all over the
filesystem.

>> [...]
>> You're crossing hard-links (file A and file B are exactly the same
>> inode, 
>
> I don't miss this point, I was saying a different thing : 
> this same inode is listed under as many directories as the 
> hardlinks

No. "inodes" don't exist in directories.  They exist in metadata tables
(e.g. stored in the first few cylinders of a FS).

"Files" in a directory are all hard-links, pointing to the inode tables.
In a lot of cases, there's only one hard-link -- such as the SINGLE file
"$HOME/test.txt" pointing to whatever inode it was assigned.

The inodes / inode tables then provide all the metadata:

  - size and physical location on disk
  - owner and group
  - permission bits
  - relevant access / modify / etc. timestamps (FS dependent)
  - reference counter of hard-links


>
>> there is no reference / dereference), and soft-links (file A
>> [inode 1234] points to fileB [inode 5678])
>
> yes to be able to cross FS boundaries
>
>> 
>>>>> to somewhere. And this huge list of items (be them inode or
>>>>> else) really slows down directory listing and scan, for
>>>>> example when I perform searches or get properties (counts,
>>>>> stats etc)
>>>>
>>>> Which'll be as equally slow, if not slower, when you try doing it on a
>>>> tarball.
>>>
>>> If one has just a huge one, for sure. That's why I wanted
>>> tarbal:folder=1:1
>> 
>> Er, which means that you have to untar the whole thing, at least into
>> memory.  Which, depending on your hardware will result in no gains, or a
>> net slowdown.
>
> lets put it from start.
>
> Long time ago in a distant galaxy, there was a file system 
> structure very deeply nested and complex, but much static in 
> its general shape and very well known and familiar to its owner
>
> The contents changed here and there.
> This wierd owner desired two different and contradictory 
> goals : to have phisical copies of the updates versions 
> (distinct files) while at the same time sharing one single 
> copy of the folder structure (or a "super-set" when some 
> change to it occurred), as in crawling in the tree he 
> seldomly required search.
> to be continued...

It sounds like you might be conflating a _filesystem_ (e.g. ext4) with a
hierarchy (e.g. the linux FHS).

A filesystem (e.g. ext4) defines how things are physically stored on
disk -- that is, stuff like where we're keeping the metadata, or how
much space we're skipping between file writes, or whether or not the
filename is directly associated to the data, etc.)

On the other hand, a hierarchy defines the layout of files and
directories on said filesystem.  On Linux FHS-compliant systems, this
means something like this:

  "/" - Root.  Everything is a child of this directory.
  "/bin" - system binaries generally required for booting
  "/boot" - bootloader / kernel 
  "/dev" - devices 
  "/etc" - short for "et cetera".  System-wide configurations are stored
    here
  "/home" - user home directories
  "/lib" - shared libraries required for booting
  "/lib[qual]" - additional shared libraries, generally used for
    multiarch support.  e.g. lib32, lib64.
  "/media" - used for automounting media
  "/mnt" - used for "temporary" mountpoints
  "/opt" - "optional" things (e.g. system-specific binaries, such as 
    an IDE or similar).[1]
  "/proc" - processor / memory / etc. hardware info
  "/root" - the home directory for the root "user"
  "/run" - runtime variable data (e.g. pidfiles, lockfiles, and so on)
  "/sbin" - "system" binaries.  Stuff that root needs for system 
    management that aren't required at boot time.
  "/srv" - data for services provided by the system (e.g. /srv/ftp)
  "/tmp" - temporary data
  "/usr" - another hierarchy of read-only system resources, generally
    provided by a distribution. It partially duplicates the root
    hierarchy with /usr/{bin,lib,sbin,share}.
  "/usr/local" - LOCAL hierarchy of read-only system resources,
    generally provided by an admin for use either on a specific host, 
    though can also mean "local to this organization".  Like /usr, 
    this contains the subdirectories bin,lib,sbin,share.

[1] Personally, I find /opt pointless, and only use it when a package
maintainer decides to dump stuff there (read: I don't wanna recompile
and put it in /usr/local/, where it belongs).

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEBcqaUD8uEzVNxUrujhHd8xJ5ooEFAlx5OF4ACgkQjhHd8xJ5
ooFfVQgAqaeOnvTSFZRFNfdcf923taXaSbrugDly7IWerYjvwEQ/T7ML1vNm8KT5
wN4fzYlSUEynqylSbAXUdf1aHMU4UAXSNg10VIa5FTqyiTu8OcOvy1yp2VUSxx9s
hhr3UP70uNU6y0q16omvTHURvHPWaIlLNr1wx8oCReIK+RMpl9yBNxD55l8wmif7
aTyppbpkXjIfVTu/jRl+CVWNAsyzhFC4zew8OUyp5Ezf8AdhUNgHNS4q38bDqSna
LvsMXV9TCsV+3DP7LxcaNlqv+th/3oetaFGde1f40oK43rTtq3Bqg4ihA32LhGOo
x7IYc31hVYsl1tYRPTd8wg+I0+s4cw==
=yoet
-----END PGP SIGNATURE-----

-- 
|_|O|_| 
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

[toc] | [prev] | [next] | [standalone]

#54272

On 01/03/2019 14.49, Dan Purgert wrote:
> Soviet_Mario wrote:
>> On 28/02/19 11:59, Dan Purgert wrote:
>>> Soviet_Mario wrote:
>>>> On 27/02/19 17:02, Dan Purgert wrote:
>>>>> Soviet_Mario wrote:
>>>>>> On 27/02/19 13:44, Aragorn wrote:

...


>> I don't miss this point, I was saying a different thing : 
>> this same inode is listed under as many directories as the 
>> hardlinks
> 
> No. "inodes" don't exist in directories.  They exist in metadata tables
> (e.g. stored in the first few cylinders of a FS).
> 
> "Files" in a directory are all hard-links, pointing to the inode tables.
> In a lot of cases, there's only one hard-link -- such as the SINGLE file
> "$HOME/test.txt" pointing to whatever inode it was assigned.
> 
> The inodes / inode tables then provide all the metadata:
> 
>   - size and physical location on disk
>   - owner and group
>   - permission bits
>   - relevant access / modify / etc. timestamps (FS dependent)
>   - reference counter of hard-links

I thought that inodes only stored the file location, but no. I did this
little test:

cer@Telcontar:~/tmp/nntp> l
total 40
drwxr-xr-x   2 cer users    25 Nov 20  2016 ./
drwxr-xr-x 106 cer users  8192 Feb 14 13:26 ../
-rw-r--r--   1 cer users 28033 Nov 20  2016 RESULTS.txt
cer@Telcontar:~/tmp/nntp> ln RESULTS.txt RESULTS2.txt
cer@Telcontar:~/tmp/nntp> l
total 68
drwxr-xr-x   2 cer users    45 Mar  1 18:17 ./
drwxr-xr-x 106 cer users  8192 Feb 14 13:26 ../
-rw-r--r--   2 cer users 28033 Nov 20  2016 RESULTS.txt
-rw-r--r--   2 cer users 28033 Nov 20  2016 RESULTS2.txt
cer@Telcontar:~/tmp/nntp> chmod a+x RESULTS2.txt
cer@Telcontar:~/tmp/nntp> l
total 68
drwxr-xr-x   2 cer users    45 Mar  1 18:17 ./
drwxr-xr-x 106 cer users  8192 Feb 14 13:26 ../
-rwxr-xr-x   2 cer users 28033 Nov 20  2016 RESULTS.txt*
-rwxr-xr-x   2 cer users 28033 Nov 20  2016 RESULTS2.txt*
cer@Telcontar:~/tmp/nntp>

Both changed permissions, which means the common block, the inode, holds
that information.


...

> [1] Personally, I find /opt pointless, and only use it when a package
> maintainer decides to dump stuff there (read: I don't wanna recompile
> and put it in /usr/local/, where it belongs).

I put there proprietary packages if I can.



-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

#54274

On Fri, 01 Mar 2019 18:22:41 +0100, Carlos E.R. scribbled:

> I thought that inodes only stored the file location, but no. I did this
> little test:
> 
> cer@Telcontar:~/tmp/nntp> l total 40 drwxr-xr-x   2 cer users    25 Nov
> 20  2016 ./
> drwxr-xr-x 106 cer users  8192 Feb 14 13:26 ../
> -rw-r--r--   1 cer users 28033 Nov 20  2016 RESULTS.txt
> cer@Telcontar:~/tmp/nntp> ln RESULTS.txt RESULTS2.txt
> cer@Telcontar:~/tmp/nntp> l total 68 drwxr-xr-x   2 cer users    45 Mar 
> 1 18:17 ./
> drwxr-xr-x 106 cer users  8192 Feb 14 13:26 ../
> -rw-r--r--   2 cer users 28033 Nov 20  2016 RESULTS.txt -rw-r--r--   2
> cer users 28033 Nov 20  2016 RESULTS2.txt cer@Telcontar:~/tmp/nntp>
> chmod a+x RESULTS2.txt cer@Telcontar:~/tmp/nntp> l total 68 drwxr-xr-x  
> 2 cer users    45 Mar  1 18:17 ./
> drwxr-xr-x 106 cer users  8192 Feb 14 13:26 ../
> -rwxr-xr-x   2 cer users 28033 Nov 20  2016 RESULTS.txt*
> -rwxr-xr-x   2 cer users 28033 Nov 20  2016 RESULTS2.txt*
> cer@Telcontar:~/tmp/nntp>
> 
> Both changed permissions, which means the common block, the inode, holds
> that information.

Yes, the permissions, owner, group, atime, mtime and ctime, link counter 
and the location of the on-disk extents with the file's actual data are 
all stored in the inode.  

For all intents and purposes, the inode _is_ the file.  The filename is 
only a directory entry (i.e. a link) pointing to it.

>> [1] Personally, I find /opt pointless, and only use it when a package
>> maintainer decides to dump stuff there (read: I don't wanna recompile
>> and put it in /usr/local/, where it belongs).
> 
> I put there proprietary packages if I can.

According to the FHS, /opt is for add-on software that's distributed in a 
binary format but that doesn't necessarily belong to the main operating 
system distribution, while /usr/local is for add-on software that has 
been compiled from sources on the local system.  

So for instance, if you were to install Firefox or LibreOffice without 
using your package manager, by using the self-extracting installers from 
their respective developers, then they should go under /opt.  And if you 
write your own software and you compile that into binaries, then they 
should go under /usr/local.

At least, that's what the FHS says.  In practice, things may be a little 
different. :)

-- 
With respect,
= Aragorn =

[toc] | [prev] | [next] | [standalone]

#54295

On 01/03/19 14:49, Dan Purgert wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> Soviet_Mario wrote:
>> On 28/02/19 11:59, Dan Purgert wrote:
>>> Soviet_Mario wrote:
>>>> On 27/02/19 17:02, Dan Purgert wrote:
>>>>> Soviet_Mario wrote:
>>>>>> On 27/02/19 13:44, Aragorn wrote:
>>>>>>> [...]
>>>>>>> rsnapshot uses hard-linking for files that haven't changed since
>>>>>>> the last backup.  Hard links don't use up any inodes.
>>>>>>>
>>>>>>
>>>>>> mmm, then I must have chosen the wrong terms.
>>>>>> But some other kind of disk space must be used to add a link
>>>>>
>>>>> Nope.
>>>>
>>>> so the link information is stored where ?
>>>> I'm not saying the content need to be stored, but the link
>>>> to it for sure : in as many places as there are the links
>>>
>>> On the filesystem/directory metadata ("in" the directory, but it doesn't
>>> take space.
>>
>> i reckon it does, even if it takes "reserved" space. And
>> anyway it takes time to be readed
> 
> You reckon wrong.  Upon creation, of the filesystem, the space is
> already used (for the inode itself).  After that, you just update the
> inode to say how many "files(tm)" are hard-linked to it.
> 
> It doesn't matter if there's only one file linked to the inode, or a
> thousand - you still have to read that inode to know where the data
> actually is.
> 
> It's faster to read a thousand "files" linked to the same inode, than
> 1000 separate inodes, as you're only reading metadata (contained in
> tightly grouped tables) rather than having to jump all over the
> filesystem.
> 
>>> [...]
>>> You're crossing hard-links (file A and file B are exactly the same
>>> inode,
>>
>> I don't miss this point, I was saying a different thing :
>> this same inode is listed under as many directories as the
>> hardlinks
> 
> No. "inodes" don't exist in directories.  They exist in metadata tables
> (e.g. stored in the first few cylinders of a FS).
> 
> "Files" in a directory are all hard-links, pointing to the inode tables.
> In a lot of cases, there's only one hard-link -- such as the SINGLE file
> "$HOME/test.txt" pointing to whatever inode it was assigned.
> 
> The inodes / inode tables then provide all the metadata:
> 
>    - size and physical location on disk
>    - owner and group
>    - permission bits
>    - relevant access / modify / etc. timestamps (FS dependent)
>    - reference counter of hard-links
> 
> 

I need to re-read that above again, seems intresting


>>
>>> there is no reference / dereference), and soft-links (file A
>>> [inode 1234] points to fileB [inode 5678])
>>
>> yes to be able to cross FS boundaries
>>
>>>
>>>>>> to somewhere. And this huge list of items (be them inode or
>>>>>> else) really slows down directory listing and scan, for
>>>>>> example when I perform searches or get properties (counts,
>>>>>> stats etc)
>>>>>
>>>>> Which'll be as equally slow, if not slower, when you try doing it on a
>>>>> tarball.
>>>>
>>>> If one has just a huge one, for sure. That's why I wanted
>>>> tarbal:folder=1:1
>>>
>>> Er, which means that you have to untar the whole thing, at least into
>>> memory.  Which, depending on your hardware will result in no gains, or a
>>> net slowdown.
>>
>> lets put it from start.
>>
>> Long time ago in a distant galaxy, there was a file system
>> structure very deeply nested and complex, but much static in
>> its general shape and very well known and familiar to its owner
>>
>> The contents changed here and there.
>> This wierd owner desired two different and contradictory
>> goals : to have phisical copies of the updates versions
>> (distinct files) while at the same time sharing one single
>> copy of the folder structure (or a "super-set" when some
>> change to it occurred), as in crawling in the tree he
>> seldomly required search.
>> to be continued...
> 
> It sounds like you might be conflating a _filesystem_ (e.g. ext4) with a
> hierarchy (e.g. the linux FHS).

I need to find some documentation about conflating a file 
system :\ Never heard about, i must admit

> 
> A filesystem (e.g. ext4) defines how things are physically stored on
> disk -- that is, stuff like where we're keeping the metadata, or how
> much space we're skipping between file writes, or whether or not the
> filename is directly associated to the data, etc.)
> 
> On the other hand, a hierarchy defines the layout of files and
> directories on said filesystem.  On Linux FHS-compliant systems, this
> means something like this:
> 
>    "/" - Root.  Everything is a child of this directory.
>    "/bin" - system binaries generally required for booting
>    "/boot" - bootloader / kernel
>    "/dev" - devices
>    "/etc" - short for "et cetera".  System-wide configurations are stored
>      here
>    "/home" - user home directories
>    "/lib" - shared libraries required for booting
>    "/lib[qual]" - additional shared libraries, generally used for
>      multiarch support.  e.g. lib32, lib64.
>    "/media" - used for automounting media
>    "/mnt" - used for "temporary" mountpoints
>    "/opt" - "optional" things (e.g. system-specific binaries, such as
>      an IDE or similar).[1]
>    "/proc" - processor / memory / etc. hardware info
>    "/root" - the home directory for the root "user"
>    "/run" - runtime variable data (e.g. pidfiles, lockfiles, and so on)
>    "/sbin" - "system" binaries.  Stuff that root needs for system
>      management that aren't required at boot time.
>    "/srv" - data for services provided by the system (e.g. /srv/ftp)
>    "/tmp" - temporary data
>    "/usr" - another hierarchy of read-only system resources, generally
>      provided by a distribution. It partially duplicates the root
>      hierarchy with /usr/{bin,lib,sbin,share}.
>    "/usr/local" - LOCAL hierarchy of read-only system resources,
>      generally provided by an admin for use either on a specific host,
>      though can also mean "local to this organization".  Like /usr,
>      this contains the subdirectories bin,lib,sbin,share.
> 
> [1] Personally, I find /opt pointless, and only use it when a package
> maintainer decides to dump stuff there (read: I don't wanna recompile
> and put it in /usr/local/, where it belongs).
> 
> -----BEGIN PGP SIGNATURE-----
> 
> iQEzBAEBCAAdFiEEBcqaUD8uEzVNxUrujhHd8xJ5ooEFAlx5OF4ACgkQjhHd8xJ5
> ooFfVQgAqaeOnvTSFZRFNfdcf923taXaSbrugDly7IWerYjvwEQ/T7ML1vNm8KT5
> wN4fzYlSUEynqylSbAXUdf1aHMU4UAXSNg10VIa5FTqyiTu8OcOvy1yp2VUSxx9s
> hhr3UP70uNU6y0q16omvTHURvHPWaIlLNr1wx8oCReIK+RMpl9yBNxD55l8wmif7
> aTyppbpkXjIfVTu/jRl+CVWNAsyzhFC4zew8OUyp5Ezf8AdhUNgHNS4q38bDqSna
> LvsMXV9TCsV+3DP7LxcaNlqv+th/3oetaFGde1f40oK43rTtq3Bqg4ihA32LhGOo
> x7IYc31hVYsl1tYRPTd8wg+I0+s4cw==
> =yoet
> -----END PGP SIGNATURE-----
> 


-- 
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
Soviet_Mario - (aka Gatto_Vizzato)

[toc] | [prev] | [next] | [standalone]

#54296

On Sun, 03 Mar 2019 01:08:55 +0100, Soviet_Mario scribbled:

> On 01/03/19 14:49, Dan Purgert wrote:
>
>> It sounds like you might be conflating a _filesystem_ (e.g. ext4) with
>> a hierarchy (e.g. the linux FHS).
> 
> I need to find some documentation about conflating a file system :\
> Never heard about, i must admit

Here you are...:

    https://www.thefreedictionary.com/conflate


[QUOTE]

   con·fla′tion n.

   Usage Note: Traditionally, conflate means "To bring together; meld or
   fuse," as in the sentence I have trouble differentiating Jane Austen's
   heroines; I realized I had conflated Elizabeth Bennet and Emma
   Woodhouse into a single character in my mind. In our 2015 survey, 87
   percent of the Usage Panelists accepted this traditional usage.
   Recently, a new sense for conflate has emerged, meaning "To mistake
   one thing for another," as if it were a synonym for confuse. In 2015,
   our usage panelists found this new sense to be marginally acceptable,
   with 55 percent accepting the sentence People often conflate the
   national debt with the federal deficit; when the senator talked about
   reducing the debt, he was actually referring to the deficit.

[/QUOTE]

:p

-- 
With respect,
= Aragorn =

[toc] | [prev] | [next] | [standalone]

#54306

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Aragorn wrote:
> On Sun, 03 Mar 2019 01:08:55 +0100, Soviet_Mario scribbled:
>
>> On 01/03/19 14:49, Dan Purgert wrote:
>>
>>> It sounds like you might be conflating a _filesystem_ (e.g. ext4) with
>>> a hierarchy (e.g. the linux FHS).
>> 
>> I need to find some documentation about conflating a file system :\
>> Never heard about, i must admit
>
> Here you are...:
>
>     https://www.thefreedictionary.com/conflate
>
>
> [QUOTE]
>
>    con·fla′tion n.
>
>    Usage Note: Traditionally, conflate means "To bring together; meld or
>    fuse,"
>    [...]
>    Recently, a new sense for conflate has emerged, meaning "To mistake
>    one thing for another," 

Note that I meant the former (proper?) definition (although that being
said, one does not simply combine a fs and a hierarchy).

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEBcqaUD8uEzVNxUrujhHd8xJ5ooEFAlx7z9kACgkQjhHd8xJ5
ooEXdQf9HcEJndacjPRoXQY6VixJS7RZK/ihuVCn31S7pIL2r0chJLXriFMMWgF7
oaaHqlFmzIy2ZK62uvydIMYneSHgSGeYoNYFGG9oYLKCqWEZBE+ZSpMSmwWYMRPL
ouFsrggiIzFjCph1TIh/4OnetBZhCrIFYOiA3Q/8KYkGgrerLd77n8i7Uyzl0hk3
AmNga4zUQyv9W8qJ9QwjXTFkyvam/4akhCp/ZFM/pd2HJEr8ovOvtxjbClFnJ9Nu
piw69lWqFV7p6ViOFraMPYvoM8zC/vEM/BcmnVLiZCB4EFJ2lrcSrzX9F2yj5GX8
lF/JyFuPWYiRp2PYgYqFt6OCCHLovA==
=8ZGb
-----END PGP SIGNATURE-----

-- 
|_|O|_| 
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

[toc] | [prev] | [next] | [standalone]

#54304

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Soviet_Mario wrote:
>> [...]
>> It sounds like you might be conflating a _filesystem_ (e.g. ext4) with a
>> hierarchy (e.g. the linux FHS).
>
> I need to find some documentation about conflating a file 
> system:\ Never heard about, i must admit

"conflate" (noun) -- combine two or more ideas into one.

There's no documentation about it, because it's not something you want
to do.

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEBcqaUD8uEzVNxUrujhHd8xJ5ooEFAlx7zKQACgkQjhHd8xJ5
ooGa3wf/YbWqEFFFWzu8cKYWXThrXdcslW3fXU+6vsm3N0t1yX7+jtnGz8+h4AEb
vkKXYCF544ZMk7kkH1TonrAGLu4nWbIkL8BLojy4bSoIl9ds0+GOJg+LJBxx4zhr
YKeXj+95NKHiUYogicuLwpoQf++snGztzqOqeKXFXJohC4oiFfwMyuEIGfiFUVSb
kCN9m/EL9en5YzqXjV+BGQNbW7nNYvqmZSQWYJjYnuroOvrgD7L+RpOB6HLLXpHX
/YCzxmCyxSmaGRYg7IiwPWAgBCHB/Mq4UDkhQmn/CRPCoBCNHDJeP6wPKgB2pwrz
vaDfhvaZjjKYmLLfrFIVP0LBy3K/GQ==
=YJKI
-----END PGP SIGNATURE-----

-- 
|_|O|_| 
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

[toc] | [prev] | [next] | [standalone]

#54262

On 2019-02-28, Dan Purgert <dan@djph.net> wrote:
>
> Soviet_Mario wrote:
>> On 27/02/19 17:02, Dan Purgert wrote:
>>> Soviet_Mario wrote:
>>>> On 27/02/19 13:44, Aragorn wrote:
>>>>> [...]
>>>>> rsnapshot uses hard-linking for files that haven't changed since
>>>>> the last backup.  Hard links don't use up any inodes.
>>>>>
>>>>
>>>> mmm, then I must have chosen the wrong terms.
>>>> But some other kind of disk space must be used to add a link
>>> 
>>> Nope.
>>
>> so the link information is stored where ?
>> I'm not saying the content need to be stored, but the link 
>> to it for sure : in as many places as there are the links
>
> On the filesystem/directory metadata ("in" the directory, but it doesn't
> take space.  Or well, worst case, it spills into the next sector of
> metadata, so burns however much space your sector size is -- generally
> somewhere between 512 and 4096 bytes).
>
>>
>> This does not duplicate the info of Inode (Aragorns taught), 
>> but then the link is a reference to be dereferenced. So when 
>> the file manager scans the disk, re-read the same inode many 
>> times or at best looks-up some hash table to find whether or 
>> not the info is just present.
>
> You're crossing hard-links (file A and file B are exactly the same
> inode, there is no reference / dereference), and soft-links (file A
> [inode 1234] points to fileB [inode 5678])
>
>>>> to somewhere. And this huge list of items (be them inode or
>>>> else) really slows down directory listing and scan, for
>>>> example when I perform searches or get properties (counts,
>>>> stats etc)
>>> 
>>> Which'll be as equally slow, if not slower, when you try doing it on a
>>> tarball.
>>
>> If one has just a huge one, for sure. That's why I wanted 
>> tarbal:folder=1:1
>
> Er, which means that you have to untar the whole thing, at least into
> memory.  Which, depending on your hardware will result in no gains, or a
> net slowdown.
>
>

[toc] | [prev] | [next] | [standalone]

#54248

On 2019-02-27, Soviet_Mario <SovietMario@CCCP.MIR> wrote:
> On 27/02/19 13:44, Aragorn wrote:
>> On Wed, 27 Feb 2019 12:52:04 +0100, Soviet_Mario scribbled:
>> 
>>> On 26/02/19 22:46, William Unruh wrote:
>>>
>>>> Something like rsnapshot (using rsync) will store about 30 backups of
>>>> 200GB of data in half a TB. (it does differential backups so only files
>>>> which changed get backed up).
>>>>
>>>> Not tarring or compressing means that the backup data is easy to get at
>>>> which is usually what you want.
>>>
>>> I call this inode space pollution, a thing that I don't find desirable
>>> on the backup target (even if I have to accept it in the userspace).
>>> It slows down searches, takes up space in itself just to store a huge
>>> number of inodes :\
>> 
>> rsnapshot uses hard-linking for files that haven't changed since the last
>> backup.  Hard links don't use up any inodes.
>> 
>
> mmm, then I must have chosen the wrong terms.
> But some other kind of disk space must be used to add a link 
> to somewhere. And this huge list of items (be them inode or 

There is of course a directory listing under each subdirectory. But
there should be. It does not slow down directory listings or scanning. 
It is just that many directory entries point to the same inode. 

> else) really slows down directory listing and scan, for 
> example when I perform searches or get properties (counts, 
> stats etc)

Why?
>

[toc] | [prev] | [next] | [standalone]

#54251

On 27/02/19 17:38, William Unruh wrote:
> On 2019-02-27, Soviet_Mario <SovietMario@CCCP.MIR> wrote:
>> On 27/02/19 13:44, Aragorn wrote:
>>> On Wed, 27 Feb 2019 12:52:04 +0100, Soviet_Mario scribbled:
>>>
>>>> On 26/02/19 22:46, William Unruh wrote:
>>>>
>>>>> Something like rsnapshot (using rsync) will store about 30 backups of
>>>>> 200GB of data in half a TB. (it does differential backups so only files
>>>>> which changed get backed up).
>>>>>
>>>>> Not tarring or compressing means that the backup data is easy to get at
>>>>> which is usually what you want.
>>>>
>>>> I call this inode space pollution, a thing that I don't find desirable
>>>> on the backup target (even if I have to accept it in the userspace).
>>>> It slows down searches, takes up space in itself just to store a huge
>>>> number of inodes :\
>>>
>>> rsnapshot uses hard-linking for files that haven't changed since the last
>>> backup.  Hard links don't use up any inodes.
>>>
>>
>> mmm, then I must have chosen the wrong terms.
>> But some other kind of disk space must be used to add a link
>> to somewhere. And this huge list of items (be them inode or
> 
> There is of course a directory listing under each subdirectory. But
> there should be. It does not slow down directory listings or scanning.
> It is just that many directory entries point to the same inode.

it does. I don't know how steep it scales up (O(n) vs 
O(n^sth) or O(log(n))), but it slows as the internally held 
cache or hash table with indexes grows as the names / links 
becomes more numerous.

I clearly suffer from this in two main contexts : 1) 
searching disk-wide. 2) scanning to get properties (like 
size and item numbers) on top-level objects.

> 
>> else) really slows down directory listing and scan, for
>> example when I perform searches or get properties (counts,
>> stats etc)
> 
> Why?

how could be different ? A static binary tree would be 
Log_2(n) in search (in the best scenario), but when the tree 
have to be created on the fly, extra time is needed anyway.

>>


-- 
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
Soviet_Mario - (aka Gatto_Vizzato)

[toc] | [prev] | [next] | [standalone]

#54257

On 2019-02-27, Soviet_Mario <SovietMario@CCCP.MIR> wrote:
> On 27/02/19 13:44, Aragorn wrote:
>> On Wed, 27 Feb 2019 12:52:04 +0100, Soviet_Mario scribbled:
>> 
>>> On 26/02/19 22:46, William Unruh wrote:
>>>
>>>> Something like rsnapshot (using rsync) will store about 30 backups of
>>>> 200GB of data in half a TB. (it does differential backups so only files
>>>> which changed get backed up).
>>>>
>>>> Not tarring or compressing means that the backup data is easy to get at
>>>> which is usually what you want.
>>>
>>> I call this inode space pollution, a thing that I don't find desirable
>>> on the backup target (even if I have to accept it in the userspace).
>>> It slows down searches, takes up space in itself just to store a huge
>>> number of inodes :\
>> 
>> rsnapshot uses hard-linking for files that haven't changed since the last
>> backup.  Hard links don't use up any inodes.
>
> mmm, then I must have chosen the wrong terms.
> But some other kind of disk space must be used to add a link 
> to somewhere.

Only the space for the directory entry, directories consume the same disk
space as file contents.

> And this huge list of items (be them inode or 
> else) really slows down directory listing and scan, for 
> example when I perform searches or get properties (counts, 
> stats etc)

You end up with a disk with lots and lots of files on it. if you scan
the whole disk that scan will be slow. other than that I have not
noticed slowness, but I've not tried using GUI tools. just command-line.



-- 
  When I tried casting out nines I made a hash of it.

[toc] | [prev] | [next] | [standalone]

#54242

On 26/02/2019 22.46, William Unruh wrote:
> On 2019-02-26, Soviet_Mario <> wrote:

...

>> yes, tnx. I located them.
>>
>> I'm using it but there is a trick to take into account if 
>> one (like me) wants to make a full backup : the /MEDIA 
>> (lowercase) folder must be excluded.
>> As I was unsure i used both the --exclude-path and --exclude-dir
>>
>> Otherwise, and i slammed the face on it, an infinite 
>> recursion takes place. It was very vierd experience to see 
>> this sort of self inclusion and reinclusion.
>>
>> Anyway (using gzip) the programs is darn thunderbolt FAST.
>>
>> By now (one third of progress) It has grinded (or grounded 
>> ?) 150 GB of compressed data size, since 20:12 to 21:48.
> 
> That would be about 1GB/min or 16MB/s which is not terribly fast. Are
> those "rust" drives or SSD?

The slowest item is probably gzip.


-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

#54235

On 2/26/19 9:51 PM, Soviet_Mario wrote:

> I'm using it but there is a trick to take into account if one (like me) 
> wants to make a full backup : the /MEDIA (lowercase) folder must be 
> excluded.
> As I was unsure i used both the --exclude-path and --exclude-dir

path usually for a specific file, while dir for a directory with all 
it's content.


> I'd really like some front end ... but well, it works and it does what I 
> needed. I notice the "compressed contentS of each folder is named 
> dbackup0.tar.gz

Then you need to look for another program to use (your current backup 
files will be most likely useless), take a look for fwbackups, bacula, 
Back In Time to mention a few.
I don't have any experience of those, as I use btrfs built in syncing to 
make incremental backups.

> I'm hoping the sw would have some hidden feature as to make 
> incremental/differential backup, generating updates like 
> dbackup1.tar.gz,  dbackup2.tar.gz etc on different runs.

I think bacula had incremental backup, please don't take my word on it, 
check it up.

-- 
   //Aho

[toc] | [prev] | [next] | [standalone]

#54241

On 26/02/2019 21.51, Soviet_Mario wrote:
> On 26/02/19 20:28, J.O. Aho wrote:


> yes, tnx. I located them.
> 
> I'm using it but there is a trick to take into account if one (like me)
> wants to make a full backup : the /MEDIA (lowercase) folder must be
> excluded.
> As I was unsure i used both the --exclude-path and --exclude-dir
> 
> Otherwise, and i slammed the face on it, an infinite recursion takes
> place. It was very vierd experience to see this sort of self inclusion
> and reinclusion.
> 
> Anyway (using gzip) the programs is darn thunderbolt FAST.
> 
> By now (one third of progress) It has grinded (or grounded ?) 150 GB of
> compressed data size, since 20:12 to 21:48.
> 
> I'd really like some front end ... but well, it works and it does what I
> needed. I notice the "compressed contentS of each folder is named
> dbackup0.tar.gz

There is a problem with tar.gz archives: a single byte error in the
backup media and the entire archive is lost.

-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

#54243

Carlos E.R. wrote:
> On 26/02/2019 21.51, Soviet_Mario wrote:
>> On 26/02/19 20:28, J.O. Aho wrote:
> 
> 
>> yes, tnx. I located them.
>>
>> I'm using it but there is a trick to take into account if one (like me)
>> wants to make a full backup : the /MEDIA (lowercase) folder must be
>> excluded.
>> As I was unsure i used both the --exclude-path and --exclude-dir
>>
>> Otherwise, and i slammed the face on it, an infinite recursion takes
>> place. It was very vierd experience to see this sort of self inclusion
>> and reinclusion.
>>
>> Anyway (using gzip) the programs is darn thunderbolt FAST.
>>
>> By now (one third of progress) It has grinded (or grounded ?) 150 GB of
>> compressed data size, since 20:12 to 21:48.
>>
>> I'd really like some front end ... but well, it works and it does what I
>> needed. I notice the "compressed contentS of each folder is named
>> dbackup0.tar.gz
> 
> There is a problem with tar.gz archives: a single byte error in the
> backup media and the entire archive is lost.
> 

You could try something like this. Perhaps par2 could
provide some redundancy for recovery.

https://en.wikipedia.org/wiki/Parchive

    Paul

[toc] | [prev] | [next] | [standalone]

#54255

On 27/02/2019 14.50, Paul wrote:
> Carlos E.R. wrote:
>> On 26/02/2019 21.51, Soviet_Mario wrote:
>>> On 26/02/19 20:28, J.O. Aho wrote:
>>
>>
>>> yes, tnx. I located them.
>>>
>>> I'm using it but there is a trick to take into account if one (like me)
>>> wants to make a full backup : the /MEDIA (lowercase) folder must be
>>> excluded.
>>> As I was unsure i used both the --exclude-path and --exclude-dir
>>>
>>> Otherwise, and i slammed the face on it, an infinite recursion takes
>>> place. It was very vierd experience to see this sort of self inclusion
>>> and reinclusion.
>>>
>>> Anyway (using gzip) the programs is darn thunderbolt FAST.
>>>
>>> By now (one third of progress) It has grinded (or grounded ?) 150 GB of
>>> compressed data size, since 20:12 to 21:48.
>>>
>>> I'd really like some front end ... but well, it works and it does what I
>>> needed. I notice the "compressed contentS of each folder is named
>>> dbackup0.tar.gz
>>
>> There is a problem with tar.gz archives: a single byte error in the
>> backup media and the entire archive is lost.
>>
> 
> You could try something like this. Perhaps par2 could
> provide some redundancy for recovery.
> 
> https://en.wikipedia.org/wiki/Parchive

Indeed! I have used it. But I would prefer using a more robust
compression format instead, not hack around it. Even getting just a
single damaged file instead of the whole lot is more acceptable.

There was one backup script that what it did was, building on cpio,
compress each file individually, not the resulting archive.

For example, rar is much more reliable. But it has another problem:
being designed for Windows, it does not support Linux filesystem
attributes. Or not all of them.


Many years ago, I used PCTools backup, for MsDOS then, and at home, with
floppies. I had two floppy drives, it would automatically use both units
automatically and write with amazing speed: so fast that I barely could
label and pack the floppies.

It used compression and forward recovery. Some years ago I tried to
recover one such backup, and it worked, even though one floppy had an
error. It worked around it. Amazing software. There is nothing like that
in Linux, tape, CD, whatever. Ease of use and solid.


So now and then I investigate compression archival programs to find one
that I like for Linux backups (reliable and with forward recovery like
par). So far, I found none that makes me happy.

So I simply use uncompressed rsync...

I'm considering on using btrfs on the backup hard disk, because I
understand it can compress it. It is the only Linux R/W transparent
filesystem. There are some for small sets, or read only. Ext2/374 has
provision for compression in the definition, but it was never
implemented AFAIK :-(


-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

Page 2 of 4 — ← Prev page 1 [2] 3 4  Next page →

Back to top | Article view | alt.os.linux

csiph-web