Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #27855 > unrolled thread

Question re alt.comp.os.linux

Back to article view | Back to comp.os.linux.misc

Contents

  Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-14 22:49 +0200
    Re: Question re alt.comp.os.linux Andreas Kohlbach <ank@spamfence.net> - 2019-08-14 17:30 -0400
      Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 00:17 +0200
        Re: Question re alt.comp.os.linux Andreas Kohlbach <ank@spamfence.net> - 2019-08-14 18:30 -0400
          Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 12:49 +0200
            Re: Question re alt.comp.os.linux wimpunk <wimpunk+news@gmail.com> - 2019-08-19 16:21 +0200
              Re: Question re alt.comp.os.linux The Natural Philosopher <tnp@invalid.invalid> - 2019-08-19 16:27 +0100
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-19 23:10 +0200
                  Re: Question re alt.comp.os.linux dillinger <dillinger@invalid.not> - 2019-08-19 23:50 +0200
                    Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-20 22:58 +0200
                      Re: Question re alt.comp.os.linux dillinger <dillinger@invalid.not> - 2019-08-21 16:37 +0200
                        Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:57 +0200
                          Re: Question re alt.comp.os.linux dillinger <dillinger@invalid.not> - 2019-08-22 07:37 +0200
                        Re: Question re alt.comp.os.linux Grant Taylor <gtaylor@tnetconsulting.net> - 2019-08-21 21:37 -0600
                          Re: Question re alt.comp.os.linux dillinger <dillinger@invalid.not> - 2019-08-22 07:35 +0200
                  Re: Question re alt.comp.os.linux Mike Easter <MikeE@ster.invalid> - 2019-08-19 15:26 -0700
                    Re: Question re alt.comp.os.linux Paul <nospam@needed.invalid> - 2019-08-19 18:52 -0400
                      Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-20 23:03 +0200
                        Re: Question re alt.comp.os.linux Paul <nospam@needed.invalid> - 2019-08-20 17:55 -0400
                          Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-21 11:32 +0200
                            Re: Question re alt.comp.os.linux Paul <nospam@needed.invalid> - 2019-08-21 07:13 -0400
                              Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:44 +0200
                            Re: Question re alt.comp.os.linux Paul <nospam@needed.invalid> - 2019-08-21 07:42 -0400
                              Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:28 +0200
                            Re: Question re alt.comp.os.linux William Unruh <unruh@invalid.ca> - 2019-08-21 15:43 +0000
                              Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:30 +0200
                        Re: Question re alt.comp.os.linux The Natural Philosopher <tnp@invalid.invalid> - 2019-08-21 17:53 +0100
                          Re: Question re alt.comp.os.linux Eli the Bearded <*@eli.users.panix.com> - 2019-08-21 18:53 +0000
                            Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:53 +0200
                              Re: Question re alt.comp.os.linux Eli the Bearded <*@eli.users.panix.com> - 2019-08-22 02:33 +0000
                                Re: Question re alt.comp.os.linux Wildman <best_lay@yahoo.com> - 2019-08-21 22:24 -0500
                                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-24 09:42 +0200
                          Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:48 +0200
                    Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-20 22:58 +0200
    Re: Question re alt.comp.os.linux "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-08-14 17:56 -0400
    Re: Question re alt.comp.os.linux Mike Easter <MikeE@ster.invalid> - 2019-08-14 15:23 -0700
      Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 12:52 +0200
        Re: Question re alt.comp.os.linux Mike Easter <MikeE@ster.invalid> - 2019-08-15 07:14 -0700
        Re: Question re alt.comp.os.linux dillinger <dillinger@invalid.not> - 2019-08-15 17:33 +0200
          Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 17:57 +0200
            Re: Question re alt.comp.os.linux jrg <jeff.g.group@att.net> - 2019-08-15 10:24 -0700
              Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 19:40 +0200
                Re: Question re alt.comp.os.linux John-Paul Stewart <jpstewart@sympatico.ca> - 2019-08-15 15:09 -0400
                  Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 21:33 +0200
                    Re: Question re alt.comp.os.linux John-Paul Stewart <jpstewart@sympatico.ca> - 2019-08-15 16:03 -0400
                      Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 22:17 +0200
              Re: Question re alt.comp.os.linux Grant Taylor <gtaylor@tnetconsulting.net> - 2019-08-15 12:53 -0600
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 21:33 +0200
            Re: Question re alt.comp.os.linux Andreas Kohlbach <ank@spamfence.net> - 2019-08-15 15:42 -0400
    Re: Question re alt.comp.os.linux Grant Taylor <gtaylor@tnetconsulting.net> - 2019-08-14 19:33 -0600
    Re: Question re alt.comp.os.linux Paul <nospam@needed.invalid> - 2019-08-14 23:52 -0400
    Re: Question re alt.comp.os.linux Chris Elvidge <chris@mshome.net> - 2019-08-15 11:35 +0200
    Re: Question re alt.comp.os.linux Peter 'Shaggy' Haywood <phaywood@alphalink.com.au> - 2019-08-31 02:15 +1000
      Re: Question re alt.comp.os.linux "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-08-30 20:24 -0400
      Re: Question re alt.comp.os.linux William Unruh <unruh@invalid.ca> - 2019-08-31 03:54 +0000
        Re: Question re alt.comp.os.linux "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-08-31 00:03 -0400
          Re: Question re alt.comp.os.linux Peter 'Shaggy' Haywood <phaywood@alphalink.com.au> - 2019-09-02 13:37 +1000
            Re: Question re alt.comp.os.linux "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-09-02 19:54 -0400
              Re: Question re alt.comp.os.linux Peter 'Shaggy' Haywood <phaywood@alphalink.com.au> - 2019-09-04 19:37 +1000
        Re: Question re alt.comp.os.linux Peter 'Shaggy' Haywood <phaywood@alphalink.com.au> - 2019-09-02 13:34 +1000

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#27889

Carlos E.R. wrote:
> On 20/08/2019 23.55, Paul wrote:
>> Carlos E.R. wrote:
>>> On 20/08/2019 00.52, Paul wrote:
>>>> Mike Easter wrote:
>>>>> Carlos E.R. wrote:
>>>>>> The messages and the group are there in
>>>>>> leafnode, but Th thinks not and insists on deleting the subscription.
>>>>> Tb has a method of clearing its index.
>>>>>
>>>>> If you are in Classic view w/ a folder pane, select the problem group
>>>>> and R click it and use the context menu for Properties.  In the window
>>>>> that comes up (Folder properties title) select the Repair folder
>>>>> button.  That will effectively delete the .msf file for the group.
>>>>>
>>>>> You will lose your pointers for the group and the msg index will be
>>>>> empty and need to be repopulated by dl/ing xx msgs (which is actually
>>>>> the msg headers only).
>>>>>
>>>>> If leafnode has those msgs and Tb and leafnode are in harmony, it
>>>>> should dl what leafnode has.
>>>>>
>>>> I think I would take a backup before punching buttons
>>>> on a thing with "a million" or "millions of files"
>>>> behind it.
>>>>
>>>> This sounds like some kind of "hard drive punishment test" :-)
>>> It is :-)
>>>
>>> I can see the hard disks grinding when I trigger a backup - a day and a
>>> half. Not of news, but everything.
>>>
>>>
>>> Once I tested several filesystems types by creating a million small
>>> files in a single directory. Only two passed: reiserfs brilliantly, and
>>> xfs (I think this got full at 800000 or so). btrfs crashed itself and
>>> crashed the machine, and before that it got veeeery slow.
>> Macrium Reflect Free does a "sequential cluster backup".
>> It makes a list, in increasing order, of every cluster that
>> makes up the file system.
> 
> <https://en.wikipedia.org/wiki/Macrium_Reflect>
> 
> «Macrium Reflect is backup utility for Microsoft Windows developed by
> Paramount Software UK Ltd in 2006. It creates disk images and file
> backup archives using Microsoft Volume Shadow Copy Service to ensure
> 'point in time' data»
> 
> 
> Remember that I use Linux?

You can boot a Macrium CD and use it to back up EXTn partitions.
I doubt BTRFS is covered.

And be aware, that this is not the emphasis of the product.
Test carefully. It did seem to work well at first (Macrium 5). I'm not
sure it's 100% today with the latest version (Macrium 7).

No matter what backup software you use, when the software
is "new to you", it should be tested.

I use this software for imaging multi-boot disks, then
putting an install set back later. (Say, five Linux test
distros and a Win10 for spice, on a GPT disk.) In one backup
run, with minimal interaction, I can back up the whole
thing in the one shot.

If a Linux partition is 20GB (out of say a 1TB partition),
you might expect an uncompress MRIMG to be around 30GB. This
could have something to do with journaling or some other
kind of structures in the file system. I don't know if this
issue (not estimating size accurately) is addressed in the
manual or not.

At one time, the Macrium CD was actually *based* on Linux,
but they stopped doing that. And that was a good call as
only some preinstall environment based on WinPE can now
safely deal with the abortion that is NTFS (unfortunately).
Macrium has actually been throwing errors, when Microsoft
changes NTFS behavior, so even this strategy is causing hair loss
for them. Microsoft is virtualizing the file system and
making companies "call an API" rather than "read structures
off disk directly" when it comes to metadata, and you know
what that means and why they're doing it.

> 
>> The advantage of this during backups, is less "random access"
>> of the disk surface during a backup.
>>
>> *******
>>
>> If you used an SSD for the million files, backup would
>> take no time at all.
> 
> 
> And do two million accesses to the ssd, wearing it out. No thank you.

Academic papers claim there is a read penalty for Flash.

If this is present in real-world devices today, it's pretty
hard to tell.

I've not seen any reports of some "read-heavy" application
"burning out" an SSD.

Writes on the other hand, the issue is quite public and
"part of the purchase". You know it has 3000 cycles or less
on writes, per location.

Note that, some early releases of TLC based SSDs had a problem
with error growth causing a reduction in read speed. And the
SSD firmware fix for this, was to basically re-write the device
every three months or so (internally). In terms of the intentions
of Flash operation, this is not exactly clever, and implies
"a bridge too far", and the Flash not living up to any kind
of performance promise (imagine you leave the SSD in a shoe
box for ten years - will it be readable? - they claim it
still meets that metric). The original intention of SSDs was
that it would pass the "10 year shoe box test". Ten years was
the original estimate for stuff like NOR flash. (And some bit rot
was noted on a few BIOS chips sitting around that long. BIOS
chips have no "sparing" or "repair".)

> 
> 
> When I want to do an efficient backup of the news store, I use a tar
> archive. That is reasonably fast and compact.
> 
>> *******
>>
>> If there are too many small files on a rotating disk drive, to make
>> for an efficient file-by-file access, you can always do a
>> "dd" backup instead. It isn't very often though, that the
>> conditions are right for that. It can turn a 10 minute
>> backup effort into a 15 hour job. However, if your backup
>> takes a day and a half anyway, then fifteen hours is an improvement :-/
> 
> Certainly, I know. It would actually take longer, as it has to backup
> empty disk space as well. Now, clonezilla would be faster, but I would
> have to pull the computer off line.  I use rsync while actually using
> the computer. I know it is not a static photo, but it is good enough.
> 
> 
> 
> My intention here was not look for a better backup strategy, but show
> the effects of a million files store, even on simple operations such as
> "delete tree".
> 

I don't know how many backup products make a list of sectors
critical to file system operation and only back those up.
As for file systems which have lots of small files, that
might be easier on the disk drive than a file-by-file approach.
Imagine if you had a 14TB drive holding 4KB files, and
random access was required of the drive during the backup.
That's a lot of head movement.

I keep an eye out for products that can do this, but don't
know of any for Linux and EXTn or other file systems.

For "dd" backups, I can make them "almost as efficient", if
I write zeros over the unused parts of the file system, then
use a simple compressor to remove sectors filled with all zeros.
Thus, a "dd" backup of 20GB of files on a 1TB partition,
takes 20GB of space (the zeroed areas being removed
from the image). The little program I wrote in C for this,
runs at around 300MB/sec (I'm doing MD5 calcs in there as well),
still not fast enough for SSDs perhaps, but good enough for
the intended hard drive targets. It's a silly idea of course,
the usage of "dd", but it *is* the method of choice when you
haven't dialed in a second backup solution yet. I can "trust"
vanilla dd, if I don't know of anything else that works
properly. Once you have developed some trust in your backup
software, you will not need "dd" all that often.

I sometimes use "dd" if I need an assurance of capturing
metadata, such as capturing the RAID metadata in the last
cylinder of a disk (connected to a "different brand" port).
No backup software is particularly tuned for something
that obscure. The last cylinder can be hidden, if the
disk drive is on a "same brand" port, like an Intel
RAID still connected to an Intel port. If I connect the
disk drive to a JMicron port and "dd" it, I get an accurate
image (capture any RAID metadata in the last cylinder).
I don't use chipset RAIDs all that often, so there isn't
a big calling for this (I only work on RAID for "reproducing
disasters" :-) ).

    Paul

[toc] | [prev] | [next] | [standalone]

#27897

On 21/08/2019 13.13, Paul wrote:
> Carlos E.R. wrote:
>> On 20/08/2019 23.55, Paul wrote:
>>> Carlos E.R. wrote:
>>>> On 20/08/2019 00.52, Paul wrote:
>>>>> Mike Easter wrote:
>>>>>> Carlos E.R. wrote:
>>>>>>> The messages and the group are there in
>>>>>>> leafnode, but Th thinks not and insists on deleting the
>>>>>>> subscription.
>>>>>> Tb has a method of clearing its index.
>>>>>>
>>>>>> If you are in Classic view w/ a folder pane, select the problem group
>>>>>> and R click it and use the context menu for Properties.  In the
>>>>>> window
>>>>>> that comes up (Folder properties title) select the Repair folder
>>>>>> button.  That will effectively delete the .msf file for the group.
>>>>>>
>>>>>> You will lose your pointers for the group and the msg index will be
>>>>>> empty and need to be repopulated by dl/ing xx msgs (which is actually
>>>>>> the msg headers only).
>>>>>>
>>>>>> If leafnode has those msgs and Tb and leafnode are in harmony, it
>>>>>> should dl what leafnode has.
>>>>>>
>>>>> I think I would take a backup before punching buttons
>>>>> on a thing with "a million" or "millions of files"
>>>>> behind it.
>>>>>
>>>>> This sounds like some kind of "hard drive punishment test" :-)
>>>> It is :-)
>>>>
>>>> I can see the hard disks grinding when I trigger a backup - a day and a
>>>> half. Not of news, but everything.
>>>>
>>>>
>>>> Once I tested several filesystems types by creating a million small
>>>> files in a single directory. Only two passed: reiserfs brilliantly, and
>>>> xfs (I think this got full at 800000 or so). btrfs crashed itself and
>>>> crashed the machine, and before that it got veeeery slow.
>>> Macrium Reflect Free does a "sequential cluster backup".
>>> It makes a list, in increasing order, of every cluster that
>>> makes up the file system.
>>
>> <https://en.wikipedia.org/wiki/Macrium_Reflect>
>>
>> «Macrium Reflect is backup utility for Microsoft Windows developed by
>> Paramount Software UK Ltd in 2006. It creates disk images and file
>> backup archives using Microsoft Volume Shadow Copy Service to ensure
>> 'point in time' data»
>>
>>
>> Remember that I use Linux?
> 
> You can boot a Macrium CD and use it to back up EXTn partitions.
> I doubt BTRFS is covered.

Not something I relish, using Windows to backup Linux :-p

I have all types of filesystems: EXTx, Reiserfs, XFS, btrfs...

> 
> And be aware, that this is not the emphasis of the product.
> Test carefully. It did seem to work well at first (Macrium 5). I'm not
> sure it's 100% today with the latest version (Macrium 7).
> 
> No matter what backup software you use, when the software
> is "new to you", it should be tested.

I'll stay then with what I know rather well :-)



> 
>>
>>> The advantage of this during backups, is less "random access"
>>> of the disk surface during a backup.
>>>
>>> *******
>>>
>>> If you used an SSD for the million files, backup would
>>> take no time at all.
>>
>>
>> And do two million accesses to the ssd, wearing it out. No thank you.
> 
> Academic papers claim there is a read penalty for Flash.
> 
> If this is present in real-world devices today, it's pretty
> hard to tell.

Dunno.

> I've not seen any reports of some "read-heavy" application
> "burning out" an SSD.
> 
> Writes on the other hand, the issue is quite public and
> "part of the purchase". You know it has 3000 cycles or less
> on writes, per location.


Telcontar:~ # smartctl -a /dev/sde | grep -i "wear\|ATTRIBUTE_NAME"
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always
      -       18
Telcontar:~ #


It uses swap heavily. It has much life left. 9500 hours.

...

>>
>> My intention here was not look for a better backup strategy, but show
>> the effects of a million files store, even on simple operations such as
>> "delete tree".
>>
> 
> I don't know how many backup products make a list of sectors
> critical to file system operation and only back those up.
> As for file systems which have lots of small files, that
> might be easier on the disk drive than a file-by-file approach.
> Imagine if you had a 14TB drive holding 4KB files, and
> random access was required of the drive during the backup.
> That's a lot of head movement.
> 
> I keep an eye out for products that can do this, but don't
> know of any for Linux and EXTn or other file systems.

partclone, used by clonezilla.

> 
> For "dd" backups, I can make them "almost as efficient", if
> I write zeros over the unused parts of the file system, then
> use a simple compressor to remove sectors filled with all zeros.
> Thus, a "dd" backup of 20GB of files on a 1TB partition,
> takes 20GB of space (the zeroed areas being removed
> from the image). The little program I wrote in C for this,
> runs at around 300MB/sec (I'm doing MD5 calcs in there as well),
> still not fast enough for SSDs perhaps, but good enough for
> the intended hard drive targets. It's a silly idea of course,
> the usage of "dd", but it *is* the method of choice when you
> haven't dialed in a second backup solution yet. I can "trust"
> vanilla dd, if I don't know of anything else that works
> properly. Once you have developed some trust in your backup
> software, you will not need "dd" all that often.
> 
> I sometimes use "dd" if I need an assurance of capturing
> metadata, such as capturing the RAID metadata in the last
> cylinder of a disk (connected to a "different brand" port).
> No backup software is particularly tuned for something
> that obscure. The last cylinder can be hidden, if the
> disk drive is on a "same brand" port, like an Intel
> RAID still connected to an Intel port. If I connect the
> disk drive to a JMicron port and "dd" it, I get an accurate
> image (capture any RAID metadata in the last cylinder).
> I don't use chipset RAIDs all that often, so there isn't
> a big calling for this (I only work on RAID for "reproducing
> disasters" :-) ).

Interesting detail.


-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

#27890

Carlos E.R. wrote:
> On 20/08/2019 23.55, Paul wrote:
>> Carlos E.R. wrote:
>>> On 20/08/2019 00.52, Paul wrote:
>>>> Mike Easter wrote:
>>>>> Carlos E.R. wrote:
>>>>>> The messages and the group are there in
>>>>>> leafnode, but Th thinks not and insists on deleting the subscription.
>>>>> Tb has a method of clearing its index.
>>>>>
>>>>> If you are in Classic view w/ a folder pane, select the problem group
>>>>> and R click it and use the context menu for Properties.  In the window
>>>>> that comes up (Folder properties title) select the Repair folder
>>>>> button.  That will effectively delete the .msf file for the group.
>>>>>
>>>>> You will lose your pointers for the group and the msg index will be
>>>>> empty and need to be repopulated by dl/ing xx msgs (which is actually
>>>>> the msg headers only).
>>>>>
>>>>> If leafnode has those msgs and Tb and leafnode are in harmony, it
>>>>> should dl what leafnode has.
>>>>>
>>>> I think I would take a backup before punching buttons
>>>> on a thing with "a million" or "millions of files"
>>>> behind it.
>>>>
>>>> This sounds like some kind of "hard drive punishment test" :-)
>>> It is :-)
>>>
>>> I can see the hard disks grinding when I trigger a backup - a day and a
>>> half. Not of news, but everything.
>>>
>>>
>>> Once I tested several filesystems types by creating a million small
>>> files in a single directory. Only two passed: reiserfs brilliantly, and
>>> xfs (I think this got full at 800000 or so). btrfs crashed itself and
>>> crashed the machine, and before that it got veeeery slow.
>> Macrium Reflect Free does a "sequential cluster backup".
>> It makes a list, in increasing order, of every cluster that
>> makes up the file system.
> 
> <https://en.wikipedia.org/wiki/Macrium_Reflect>
> 
> «Macrium Reflect is backup utility for Microsoft Windows developed by
> Paramount Software UK Ltd in 2006. It creates disk images and file
> backup archives using Microsoft Volume Shadow Copy Service to ensure
> 'point in time' data»
> 
> 
> Remember that I use Linux?

I just found this in a search.

So there is an open source program with the same approachas Macrium.
(For all I know, the Macrium routine could be using this.)

https://linuxconfig.org/how-to-use-partclone-to-create-a-smart-partition-backup

The requirement is to not have the partition mounted at
the time the backup is made. The "smart" part, is only
recording the sectors of the disk that hold critical
information. As long as the list of sectors/inodes/clusters
is known in advance, you can move the heads smoothly across
the disk surface during the backup. That's the potential
advantage if a partition had an extreme number of files on it.

    Paul

[toc] | [prev] | [next] | [standalone]

#27896

On 21/08/2019 13.42, Paul wrote:
> Carlos E.R. wrote:
>> On 20/08/2019 23.55, Paul wrote:


>> <https://en.wikipedia.org/wiki/Macrium_Reflect>
>>
>> «Macrium Reflect is backup utility for Microsoft Windows developed by
>> Paramount Software UK Ltd in 2006. It creates disk images and file
>> backup archives using Microsoft Volume Shadow Copy Service to ensure
>> 'point in time' data»
>>
>>
>> Remember that I use Linux?
> 
> I just found this in a search.
> 
> So there is an open source program with the same approachas Macrium.
> (For all I know, the Macrium routine could be using this.)
> 
> https://linuxconfig.org/how-to-use-partclone-to-create-a-smart-partition-backup
> 
> 

Of course, clonezilla uses it. That's what I use for offline imaging :-)

Normally, I do a full dd of the Windows partitions, and an rsync backup
of Linux partitions, plus enough information to restore boot areas and
partitions. I have all that tied into a script that runs off a bootable
external hard disk, that can be encrypted. The last one I created is
also compressed (btrfs), it is a compressed rsync copy. Very efficient.

Full dd is more reliable for restoring that partclone's. Not as
efficient, but fully reliable. Unless you will change destination
partition size. In fact, currently clonezilla uses dd on ntfs
partitions. At least the copy I have does that.

-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

#27892

On 2019-08-21, Carlos E.R. <robin_listas@es.invalid> wrote:
> On 20/08/2019 23.55, Paul wrote:
....
>> *******
>> 
>> If you used an SSD for the million files, backup would
>> take no time at all.
>
>
> And do two million accesses to the ssd, wearing it out. No thank you.

Uh, ONLY erase "wears it out". You can do as many reads as you want. 
Note that both reads and erases (including writes) "wear out" a spinning
rust disk, and the torture of that head moving back and forth wears it
out as well as the memory.

[toc] | [prev] | [next] | [standalone]

#27895

On 21/08/2019 17.43, William Unruh wrote:
> On 2019-08-21, Carlos E.R. <robin_listas@es.invalid> wrote:
>> On 20/08/2019 23.55, Paul wrote:
> ....
>>> *******
>>>
>>> If you used an SSD for the million files, backup would
>>> take no time at all.
>>
>>
>> And do two million accesses to the ssd, wearing it out. No thank you.
> 
> Uh, ONLY erase "wears it out". You can do as many reads as you want. 
> Note that both reads and erases (including writes) "wear out" a spinning
> rust disk, and the torture of that head moving back and forth wears it
> out as well as the memory.
> 

Certainly.

But the assumption is, an SSD on the source store, and another on the
destination (backup) store.

-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

#27893

On 20/08/2019 22:03, Carlos E.R. wrote:
> Once I tested several filesystems types by creating a million small
> files in a single directory. Only two passed: reiserfs brilliantly, and
> xfs (I think this got full at 800000 or so). btrfs crashed itself and
> crashed the machine, and before that it got veeeery slow.

I'll remember that if I ever want to do something as staggeringingly 
unhelpful as that

MySQL was designed for that..


-- 
New Socialism consists essentially in being seen to have your heart in 
the right place whilst your head is in the clouds and your hand is in 
someone else's pocket.

[toc] | [prev] | [next] | [standalone]

#27894

In comp.os.linux.misc, The Natural Philosopher  <tnp@invalid.invalid> wrote:
> On 20/08/2019 22:03, Carlos E.R. wrote:
>> Once I tested several filesystems types by creating a million small
>> files in a single directory. Only two passed: reiserfs brilliantly, and
>> xfs (I think this got full at 800000 or so). btrfs crashed itself and
>> crashed the machine, and before that it got veeeery slow.

Seems to me ext4 works decently at that, as well as the Netapp native
one. I've had real world programs that have tested filesystems like
this.

One of them I had to switch from ext3 to xfs because while ext3 could
handle the filecount, it couldn't handle the subdirectory count.  (This
was a Perforce server, about 2012. Perforce uses subdirectories for what
it calls "views"; in git terms perhaps a "checkout" corresponds. Some of
my users used a lot of views during code automation work.)

Another one was an opensource program to create mosaics from a
collection of images. It created thumbnails of all the source images,
the thumbnails would then be stitched together to crete the mosaic. But
it put all of the thumbnails in one directory. As I noticed *after* I
ran some 300k images through it.

> I'll remember that if I ever want to do something as staggeringingly 
> unhelpful as that

Real world involves dealing with programs at scales the original
designers perhaps had not considered.

> MySQL was designed for that..

Mysql is a piece of crap, and I'd trust a filesystem to return better
errors when things fail.

Elijah
------
had access to a huge set of media images at the time

[toc] | [prev] | [next] | [standalone]

#27899

On 21/08/2019 20.53, Eli the Bearded wrote:
> In comp.os.linux.misc, The Natural Philosopher  <tnp@invalid.invalid> wrote:
>> On 20/08/2019 22:03, Carlos E.R. wrote:
>>> Once I tested several filesystems types by creating a million small
>>> files in a single directory. Only two passed: reiserfs brilliantly, and
>>> xfs (I think this got full at 800000 or so). btrfs crashed itself and
>>> crashed the machine, and before that it got veeeery slow.
> 
> Seems to me ext4 works decently at that, as well as the Netapp native
> one. I've had real world programs that have tested filesystems like
> this.

Ext3/4 has a limited number of inodes, so a limited number of files.

> One of them I had to switch from ext3 to xfs because while ext3 could
> handle the filecount, it couldn't handle the subdirectory count.  (This
> was a Perforce server, about 2012. Perforce uses subdirectories for what
> it calls "views"; in git terms perhaps a "checkout" corresponds. Some of
> my users used a lot of views during code automation work.)
> 
> Another one was an opensource program to create mosaics from a
> collection of images. It created thumbnails of all the source images,
> the thumbnails would then be stitched together to crete the mosaic. But
> it put all of the thumbnails in one directory. As I noticed *after* I
> ran some 300k images through it.

Those are the things on which reiserfs worked brilliantly. Pity that its
main developer committed a crime and development stalled. XFS works
quite well because it doesn't have a fixed number of inodes, as the EXT
family, so it is the practical alternative. Btrfs... I don't know.

> 
>> I'll remember that if I ever want to do something as staggeringingly 
>> unhelpful as that
> 
> Real world involves dealing with programs at scales the original
> designers perhaps had not considered.
> 
>> MySQL was designed for that..
> 
> Mysql is a piece of crap, and I'd trust a filesystem to return better
> errors when things fail.

Quite...

And of course, you can easily access a filesystem from a script.

-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

#27901

In comp.os.linux.misc, Carlos E.R. <robin_listas@es.invalid> wrote:
> On 21/08/2019 20.53, Eli the Bearded wrote:
> Ext3/4 has a limited number of inodes, so a limited number of files.

Does it? I thought it fixed that and it will continue allocating them as
needed until disk runs out.

> Those are the things on which reiserfs worked brilliantly. Pity that its
> main developer committed a crime and development stalled.

Reiserfs, however, was pretty buggy with fsck of volumes that contain
reiserfs disk images, as you might have on hand if you do any sort of
virtualization. Bugs like that I think helped kill it off quickly once
Hans "Wife Murderer" Reiser stopped working on it.

> XFS works quite well because it doesn't have a fixed number of inodes,

XFS was, then and might still be, the recommended Linux filesystem for
Perforce.

> And of course, you can easily access a filesystem from a script.

You can access databases from scripts, too, but there is an added amount
of friction involved.

Elijah
------
has scripts to access mysql and pgsql databases

[toc] | [prev] | [next] | [standalone]

#27902

On Thu, 22 Aug 2019 02:33:27 +0000, Eli the Bearded wrote:

> In comp.os.linux.misc, Carlos E.R. <robin_listas@es.invalid> wrote:
>> On 21/08/2019 20.53, Eli the Bearded wrote:
>> Ext3/4 has a limited number of inodes, so a limited number of files.
> 
> Does it? I thought it fixed that and it will continue allocating them as
> needed until disk runs out.

EXT2/3/4 holds inode numbers in a 32-bit structure so the maximum
number is 2^32 or around 4 billion.  The actual limit is set in
/etc/mke2fs.conf using the inode_ratio setting.  A reformat is
required for a change.  It is not changed on the fly.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!

[toc] | [prev] | [next] | [standalone]

#27947

On 22/08/2019 04.33, Eli the Bearded wrote:
> In comp.os.linux.misc, Carlos E.R. <robin_listas@es.invalid> wrote:
>> On 21/08/2019 20.53, Eli the Bearded wrote:
>> Ext3/4 has a limited number of inodes, so a limited number of files.
> 
> Does it? I thought it fixed that and it will continue allocating them as
> needed until disk runs out.
> 
>> Those are the things on which reiserfs worked brilliantly. Pity that its
>> main developer committed a crime and development stalled.
> 
> Reiserfs, however, was pretty buggy with fsck of volumes that contain
> reiserfs disk images, as you might have on hand if you do any sort of
> virtualization. 

I think this one was solved long ago.

> Bugs like that I think helped kill it off quickly once
> Hans "Wife Murderer" Reiser stopped working on it.
> 
>> XFS works quite well because it doesn't have a fixed number of inodes,
> 
> XFS was, then and might still be, the recommended Linux filesystem for
> Perforce.

It is the default on openSUSE for /home partition.

> 
>> And of course, you can easily access a filesystem from a script.
> 
> You can access databases from scripts, too, but there is an added amount
> of friction involved.
> 
> Elijah
> ------
> has scripts to access mysql and pgsql databases
> 


-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

#27898

On 21/08/2019 18.53, The Natural Philosopher wrote:
> On 20/08/2019 22:03, Carlos E.R. wrote:
>> Once I tested several filesystems types by creating a million small
>> files in a single directory. Only two passed: reiserfs brilliantly, and
>> xfs (I think this got full at 800000 or so). btrfs crashed itself and
>> crashed the machine, and before that it got veeeery slow.
> 
> I'll remember that if I ever want to do something as staggeringingly
> unhelpful as that
> 
> MySQL was designed for that..

And reiserfs was designed to hold databases directly, a file for each
record, if wanted. Why invent a database engine, when a filesystem can
do it better and faster?  :-p

Specially version 4.


Who needs a million files in a single directory? Maildir does. A mail
folder with a million posts. :-P


-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

#27884

On 20/08/2019 00.26, Mike Easter wrote:
> Carlos E.R. wrote:
>> The messages and the group are there in
>> leafnode, but Th thinks not and insists on deleting the subscription.
> 
> Tb has a method of clearing its index.
> 
> If you are in Classic view w/ a folder pane, select the problem group
> and R click it and use the context menu for Properties.  In the window
> that comes up (Folder properties title) select the Repair folder button.
>  That will effectively delete the .msf file for the group.

I know.

But it keeps track (fortunately) of what has been read. Most times.

> 
> You will lose your pointers for the group and the msg index will be
> empty and need to be repopulated by dl/ing xx msgs (which is actually
> the msg headers only).
> 
> If leafnode has those msgs and Tb and leafnode are in harmony, it should
> dl what leafnode has.

No, because TB no longer sees this two groups. Impossible to repair what
it does not see.

-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

#27857

On Wed, 14 Aug 2019 16:49:55 -0400, Carlos E.R. <robin_listas@es.invalid> wrote:

> Since some days ago Thunderbird insists that the groups
> alt.comp.os.linux (and alt.comp.lang.pascal) do not exist and wants to
> unsubscribe from them. I hit cancel, but after a while it asks again.
> And the directory /var/spool/news/alt/comp/os/linux is populated with
> old messages (I use leafnode), so the group does exist on the server.

Try forcing tb to refresh the list of newsgroups.
- In the left panel, click on the leafnode server
- In the right panel in the top section click on Manage newsgroups subscriptions.
- Click on the refresh button on the right side of the dialog.

Regards, Dave Hodgins

-- 
Change dwhodgins@nomail.afraid.org to davidwhodgins@teksavvy.com for
email replies.

[toc] | [prev] | [next] | [standalone]

#27859

Carlos E.R. wrote:
> 2) Have those groups really disappeared?

My NSP does not carry alt.comp.lang.pascal or alt.comp.os.linux.

If I were to refresh my groups list, my NSP would provide me w/ a list 
of the groups which it carries; but your situation is different.

You via Tb 'correspond' w/ Leafnode, not a regular NSP.  Your Leafnode 
gets its feeds from some feed.

The leafnode docs talk about problems between fetchnews (and OE) but not 
specifically about Tb.  This fetchnews item is of some interest:

// 6. I have unsubscribed from a newsgroup, but fetchnews still pulls 
articles for
    that group.

    Your news reader talks to leafnode via the NNTP protocol. This protocol
    provides no means for Leafnode to determine which newsgroups you are
    actually subscribe. Therefore, Leafnode assumes that a newsgroup 
that is not
    read for a certain time (which can be configured with the timeout_long
    parameter) is unsubscribed and will only stop retrieving articles in it
    after this time.

    If you are impatient and want to stop retrieving articles from that 
group
    immediately, delete the corresponding file in the /var/spool/news/
    interesting.groups/ directory. The articles that are already in your 
spool
    are still subject to the regular texpire schedule, however.
//



-- 
Mike Easter

[toc] | [prev] | [next] | [standalone]

#27865

On 15/08/2019 00.23, Mike Easter wrote:
> Carlos E.R. wrote:
>> 2) Have those groups really disappeared?
> 
> My NSP does not carry alt.comp.lang.pascal or alt.comp.os.linux.
> 
> If I were to refresh my groups list, my NSP would provide me w/ a list
> of the groups which it carries; but your situation is different.
> 
> You via Tb 'correspond' w/ Leafnode, not a regular NSP.  Your Leafnode
> gets its feeds from some feed.

Correct. From Individual.NET (pay) and aioe.org as backup.

> 
> The leafnode docs talk about problems between fetchnews (and OE) but not
> specifically about Tb.  This fetchnews item is of some interest:
> 
> // 6. I have unsubscribed from a newsgroup, but fetchnews still pulls
> articles for
>    that group.
> 
>    Your news reader talks to leafnode via the NNTP protocol. This protocol
>    provides no means for Leafnode to determine which newsgroups you are
>    actually subscribe. Therefore, Leafnode assumes that a newsgroup that
> is not
>    read for a certain time (which can be configured with the timeout_long
>    parameter) is unsubscribed and will only stop retrieving articles in it
>    after this time.

Right. Yes, I know this mechanism. But I can not do this as those two
groups are no longer displayed by TB, yet it nags to unsubscribe from them.

> 
>    If you are impatient and want to stop retrieving articles from that
> group
>    immediately, delete the corresponding file in the /var/spool/news/
>    interesting.groups/ directory. The articles that are already in your
> spool
>    are still subject to the regular texpire schedule, however.
> //
> 
> 
> 


-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

#27866

Carlos E.R. wrote:
> Mike Easter wrote:
>> Carlos E.R. wrote:
>>> 2) Have those groups really disappeared?
>>
>> My NSP does not carry alt.comp.lang.pascal or alt.comp.os.linux.
>>
>> If I were to refresh my groups list, my NSP would provide me w/ a list
>> of the groups which it carries; but your situation is different.
>>
>> You via Tb 'correspond' w/ Leafnode, not a regular NSP.  Your Leafnode
>> gets its feeds from some feed.
> 
> Correct. From Individual.NET (pay) and aioe.org as backup.
> 
>>
>> The leafnode docs talk about problems between fetchnews (and OE) but not
>> specifically about Tb.  This fetchnews item is of some interest:
>>
>> // 6. I have unsubscribed from a newsgroup, but fetchnews still pulls
>> articles for
>>     that group.
>>
>>     Your news reader talks to leafnode via the NNTP protocol. This protocol
>>     provides no means for Leafnode to determine which newsgroups you are
>>     actually subscribe. Therefore, Leafnode assumes that a newsgroup that
>> is not
>>     read for a certain time (which can be configured with the timeout_long
>>     parameter) is unsubscribed and will only stop retrieving articles in it
>>     after this time.
> 
> Right. Yes, I know this mechanism. But I can not do this as those two
> groups are no longer displayed by TB, yet it nags to unsubscribe from them.

I think you should remove those 2 groups from leafnode's interesting groups.

I think here:  "opt/leafnode-1/var/spool/news/interesting.groups 
contains one file for each group an NNTP client has asked to read."

>>     If you are impatient and want to stop retrieving articles from that
>> group
>>     immediately, delete the corresponding file in the /var/spool/news/
>>     interesting.groups/ directory. The articles that are already in your
>> spool
>>     are still subject to the regular texpire schedule, however.
>> //



-- 
Mike Easter

[toc] | [prev] | [next] | [standalone]

#27867

On 8/15/19 12:52 PM, Carlos E.R. wrote:

> On 15/08/2019 00.23, Mike Easter wrote:
>> Carlos E.R. wrote:
>>> 2) Have those groups really disappeared?
>> My NSP does not carry alt.comp.lang.pascal or alt.comp.os.linux.
>>
>> If I were to refresh my groups list, my NSP would provide me w/ a list
>> of the groups which it carries; but your situation is different.
>>
>> You via Tb 'correspond' w/ Leafnode, not a regular NSP.  Your Leafnode
>> gets its feeds from some feed.
> Correct. From Individual.NET (pay) and aioe.org as backup.
> 

I can't tell about individual.net but it looks like aioe.org still 
carries those groups:

headers dated August 6 & 2:

Path:
 
aioe.org!news.dns-netz.com!news.freedyn.net!newsreader4.netcologne.de!news.
	netcologne.de!peer03.ams1!peer.ams1.xlned.com!news.xlned.com!peer03.am4!peer.am4.highwinds-media.com!peer02.iad!feed-me.highwinds-media.com!news.highwinds-media.com!spln!extra.newsguy.com!newsp.newsguy.com!news3
From: Shadow <Sh@dow.br>
Newsgroups: alt.comp.lang.pascal,alt.comp.freeware
Subject: Lazarus 2.0.4 released
Date: Tue, 06 Aug 2019 10:56:28 -0300

Path:
 
aioe.org!feeder.usenetexpress.com!feeder-in1.iad1.usenetexpress.com!border1
	.nntp.dca1.giganews.com!border2.nntp.dca1.giganews.com!nntp.giganews.com!b26no5697203qtq.0!news-out.google.com!a5ni933qtd.0!nntp.google.com!b26no5697188qtq.0!postnews.google.com!glegroupsg2000goo.googlegroups.com!not-for-mail
Newsgroups: alt.comp.os.linux
Date: Fri, 2 Aug 2019 08:06:55 -0700 (PDT)

Something must be wrong with your leafnode, you can try fetchnews -f

[toc] | [prev] | [next] | [standalone]

#27868

On 15/08/2019 17.33, dillinger wrote:
> On 8/15/19 12:52 PM, Carlos E.R. wrote:
> 
>> On 15/08/2019 00.23, Mike Easter wrote:
>>> Carlos E.R. wrote:
>>>> 2) Have those groups really disappeared?
>>> My NSP does not carry alt.comp.lang.pascal or alt.comp.os.linux.
>>>
>>> If I were to refresh my groups list, my NSP would provide me w/ a list
>>> of the groups which it carries; but your situation is different.
>>>
>>> You via Tb 'correspond' w/ Leafnode, not a regular NSP.  Your Leafnode
>>> gets its feeds from some feed.
>> Correct. From Individual.NET (pay) and aioe.org as backup.
>>
> 
> I can't tell about individual.net but it looks like aioe.org still
> carries those groups:
> 
> headers dated August 6 & 2:
> 
> Path:
> 
> aioe.org!news.dns-netz.com!news.freedyn.net!newsreader4.netcologne.de!news.
>     netcologne.de!peer03.ams1!peer.ams1.xlned.com!news.xlned.com!peer03.am4!peer.am4.highwinds-media.com!peer02.iad!feed-me.highwinds-media.com!news.highwinds-media.com!spln!extra.newsguy.com!newsp.newsguy.com!news3
> 
> From: Shadow <Sh@dow.br>
> Newsgroups: alt.comp.lang.pascal,alt.comp.freeware
> Subject: Lazarus 2.0.4 released
> Date: Tue, 06 Aug 2019 10:56:28 -0300
> 
> Path:
> 
> aioe.org!feeder.usenetexpress.com!feeder-in1.iad1.usenetexpress.com!border1
>     .nntp.dca1.giganews.com!border2.nntp.dca1.giganews.com!nntp.giganews.com!b26no5697203qtq.0!news-out.google.com!a5ni933qtd.0!nntp.google.com!b26no5697188qtq.0!postnews.google.com!glegroupsg2000goo.googlegroups.com!not-for-mail
> 
> Newsgroups: alt.comp.os.linux
> Date: Fri, 2 Aug 2019 08:06:55 -0700 (PDT)
> 
> Something must be wrong with your leafnode, you can try fetchnews -f

I know, something is wrong.

-- 
Cheers, Carlos.

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.os.linux.misc

csiph-web