Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #27897

Re: Question re alt.comp.os.linux

Cross-posted to 2 groups.

Show all headers | View raw

On 21/08/2019 13.13, Paul wrote:
> Carlos E.R. wrote:
>> On 20/08/2019 23.55, Paul wrote:
>>> Carlos E.R. wrote:
>>>> On 20/08/2019 00.52, Paul wrote:
>>>>> Mike Easter wrote:
>>>>>> Carlos E.R. wrote:
>>>>>>> The messages and the group are there in
>>>>>>> leafnode, but Th thinks not and insists on deleting the
>>>>>>> subscription.
>>>>>> Tb has a method of clearing its index.
>>>>>>
>>>>>> If you are in Classic view w/ a folder pane, select the problem group
>>>>>> and R click it and use the context menu for Properties.  In the
>>>>>> window
>>>>>> that comes up (Folder properties title) select the Repair folder
>>>>>> button.  That will effectively delete the .msf file for the group.
>>>>>>
>>>>>> You will lose your pointers for the group and the msg index will be
>>>>>> empty and need to be repopulated by dl/ing xx msgs (which is actually
>>>>>> the msg headers only).
>>>>>>
>>>>>> If leafnode has those msgs and Tb and leafnode are in harmony, it
>>>>>> should dl what leafnode has.
>>>>>>
>>>>> I think I would take a backup before punching buttons
>>>>> on a thing with "a million" or "millions of files"
>>>>> behind it.
>>>>>
>>>>> This sounds like some kind of "hard drive punishment test" :-)
>>>> It is :-)
>>>>
>>>> I can see the hard disks grinding when I trigger a backup - a day and a
>>>> half. Not of news, but everything.
>>>>
>>>>
>>>> Once I tested several filesystems types by creating a million small
>>>> files in a single directory. Only two passed: reiserfs brilliantly, and
>>>> xfs (I think this got full at 800000 or so). btrfs crashed itself and
>>>> crashed the machine, and before that it got veeeery slow.
>>> Macrium Reflect Free does a "sequential cluster backup".
>>> It makes a list, in increasing order, of every cluster that
>>> makes up the file system.
>>
>> <https://en.wikipedia.org/wiki/Macrium_Reflect>
>>
>> «Macrium Reflect is backup utility for Microsoft Windows developed by
>> Paramount Software UK Ltd in 2006. It creates disk images and file
>> backup archives using Microsoft Volume Shadow Copy Service to ensure
>> 'point in time' data»
>>
>>
>> Remember that I use Linux?
> 
> You can boot a Macrium CD and use it to back up EXTn partitions.
> I doubt BTRFS is covered.

Not something I relish, using Windows to backup Linux :-p

I have all types of filesystems: EXTx, Reiserfs, XFS, btrfs...

> 
> And be aware, that this is not the emphasis of the product.
> Test carefully. It did seem to work well at first (Macrium 5). I'm not
> sure it's 100% today with the latest version (Macrium 7).
> 
> No matter what backup software you use, when the software
> is "new to you", it should be tested.

I'll stay then with what I know rather well :-)



> 
>>
>>> The advantage of this during backups, is less "random access"
>>> of the disk surface during a backup.
>>>
>>> *******
>>>
>>> If you used an SSD for the million files, backup would
>>> take no time at all.
>>
>>
>> And do two million accesses to the ssd, wearing it out. No thank you.
> 
> Academic papers claim there is a read penalty for Flash.
> 
> If this is present in real-world devices today, it's pretty
> hard to tell.

Dunno.

> I've not seen any reports of some "read-heavy" application
> "burning out" an SSD.
> 
> Writes on the other hand, the issue is quite public and
> "part of the purchase". You know it has 3000 cycles or less
> on writes, per location.


Telcontar:~ # smartctl -a /dev/sde | grep -i "wear\|ATTRIBUTE_NAME"
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always
      -       18
Telcontar:~ #


It uses swap heavily. It has much life left. 9500 hours.

...

>>
>> My intention here was not look for a better backup strategy, but show
>> the effects of a million files store, even on simple operations such as
>> "delete tree".
>>
> 
> I don't know how many backup products make a list of sectors
> critical to file system operation and only back those up.
> As for file systems which have lots of small files, that
> might be easier on the disk drive than a file-by-file approach.
> Imagine if you had a 14TB drive holding 4KB files, and
> random access was required of the drive during the backup.
> That's a lot of head movement.
> 
> I keep an eye out for products that can do this, but don't
> know of any for Linux and EXTn or other file systems.

partclone, used by clonezilla.

> 
> For "dd" backups, I can make them "almost as efficient", if
> I write zeros over the unused parts of the file system, then
> use a simple compressor to remove sectors filled with all zeros.
> Thus, a "dd" backup of 20GB of files on a 1TB partition,
> takes 20GB of space (the zeroed areas being removed
> from the image). The little program I wrote in C for this,
> runs at around 300MB/sec (I'm doing MD5 calcs in there as well),
> still not fast enough for SSDs perhaps, but good enough for
> the intended hard drive targets. It's a silly idea of course,
> the usage of "dd", but it *is* the method of choice when you
> haven't dialed in a second backup solution yet. I can "trust"
> vanilla dd, if I don't know of anything else that works
> properly. Once you have developed some trust in your backup
> software, you will not need "dd" all that often.
> 
> I sometimes use "dd" if I need an assurance of capturing
> metadata, such as capturing the RAID metadata in the last
> cylinder of a disk (connected to a "different brand" port).
> No backup software is particularly tuned for something
> that obscure. The last cylinder can be hidden, if the
> disk drive is on a "same brand" port, like an Intel
> RAID still connected to an Intel port. If I connect the
> disk drive to a JMicron port and "dd" it, I get an accurate
> image (capture any RAID metadata in the last cylinder).
> I don't use chipset RAIDs all that often, so there isn't
> a big calling for this (I only work on RAID for "reproducing
> disasters" :-) ).

Interesting detail.


-- 
Cheers, Carlos.

Back to comp.os.linux.misc | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-14 22:49 +0200
  Re: Question re alt.comp.os.linux Andreas Kohlbach <ank@spamfence.net> - 2019-08-14 17:30 -0400
    Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 00:17 +0200
      Re: Question re alt.comp.os.linux Andreas Kohlbach <ank@spamfence.net> - 2019-08-14 18:30 -0400
        Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 12:49 +0200
          Re: Question re alt.comp.os.linux wimpunk <wimpunk+news@gmail.com> - 2019-08-19 16:21 +0200
            Re: Question re alt.comp.os.linux The Natural Philosopher <tnp@invalid.invalid> - 2019-08-19 16:27 +0100
              Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-19 23:10 +0200
                Re: Question re alt.comp.os.linux dillinger <dillinger@invalid.not> - 2019-08-19 23:50 +0200
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-20 22:58 +0200
                Re: Question re alt.comp.os.linux dillinger <dillinger@invalid.not> - 2019-08-21 16:37 +0200
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:57 +0200
                Re: Question re alt.comp.os.linux dillinger <dillinger@invalid.not> - 2019-08-22 07:37 +0200
                Re: Question re alt.comp.os.linux Grant Taylor <gtaylor@tnetconsulting.net> - 2019-08-21 21:37 -0600
                Re: Question re alt.comp.os.linux dillinger <dillinger@invalid.not> - 2019-08-22 07:35 +0200
                Re: Question re alt.comp.os.linux Mike Easter <MikeE@ster.invalid> - 2019-08-19 15:26 -0700
                Re: Question re alt.comp.os.linux Paul <nospam@needed.invalid> - 2019-08-19 18:52 -0400
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-20 23:03 +0200
                Re: Question re alt.comp.os.linux Paul <nospam@needed.invalid> - 2019-08-20 17:55 -0400
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-21 11:32 +0200
                Re: Question re alt.comp.os.linux Paul <nospam@needed.invalid> - 2019-08-21 07:13 -0400
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:44 +0200
                Re: Question re alt.comp.os.linux Paul <nospam@needed.invalid> - 2019-08-21 07:42 -0400
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:28 +0200
                Re: Question re alt.comp.os.linux William Unruh <unruh@invalid.ca> - 2019-08-21 15:43 +0000
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:30 +0200
                Re: Question re alt.comp.os.linux The Natural Philosopher <tnp@invalid.invalid> - 2019-08-21 17:53 +0100
                Re: Question re alt.comp.os.linux Eli the Bearded <*@eli.users.panix.com> - 2019-08-21 18:53 +0000
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:53 +0200
                Re: Question re alt.comp.os.linux Eli the Bearded <*@eli.users.panix.com> - 2019-08-22 02:33 +0000
                Re: Question re alt.comp.os.linux Wildman <best_lay@yahoo.com> - 2019-08-21 22:24 -0500
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-24 09:42 +0200
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-22 00:48 +0200
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-20 22:58 +0200
  Re: Question re alt.comp.os.linux "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-08-14 17:56 -0400
  Re: Question re alt.comp.os.linux Mike Easter <MikeE@ster.invalid> - 2019-08-14 15:23 -0700
    Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 12:52 +0200
      Re: Question re alt.comp.os.linux Mike Easter <MikeE@ster.invalid> - 2019-08-15 07:14 -0700
      Re: Question re alt.comp.os.linux dillinger <dillinger@invalid.not> - 2019-08-15 17:33 +0200
        Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 17:57 +0200
          Re: Question re alt.comp.os.linux jrg <jeff.g.group@att.net> - 2019-08-15 10:24 -0700
            Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 19:40 +0200
              Re: Question re alt.comp.os.linux John-Paul Stewart <jpstewart@sympatico.ca> - 2019-08-15 15:09 -0400
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 21:33 +0200
                Re: Question re alt.comp.os.linux John-Paul Stewart <jpstewart@sympatico.ca> - 2019-08-15 16:03 -0400
                Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 22:17 +0200
            Re: Question re alt.comp.os.linux Grant Taylor <gtaylor@tnetconsulting.net> - 2019-08-15 12:53 -0600
              Re: Question re alt.comp.os.linux "Carlos E.R." <robin_listas@es.invalid> - 2019-08-15 21:33 +0200
          Re: Question re alt.comp.os.linux Andreas Kohlbach <ank@spamfence.net> - 2019-08-15 15:42 -0400
  Re: Question re alt.comp.os.linux Grant Taylor <gtaylor@tnetconsulting.net> - 2019-08-14 19:33 -0600
  Re: Question re alt.comp.os.linux Paul <nospam@needed.invalid> - 2019-08-14 23:52 -0400
  Re: Question re alt.comp.os.linux Chris Elvidge <chris@mshome.net> - 2019-08-15 11:35 +0200
  Re: Question re alt.comp.os.linux Peter 'Shaggy' Haywood <phaywood@alphalink.com.au> - 2019-08-31 02:15 +1000
    Re: Question re alt.comp.os.linux "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-08-30 20:24 -0400
    Re: Question re alt.comp.os.linux William Unruh <unruh@invalid.ca> - 2019-08-31 03:54 +0000
      Re: Question re alt.comp.os.linux "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-08-31 00:03 -0400
        Re: Question re alt.comp.os.linux Peter 'Shaggy' Haywood <phaywood@alphalink.com.au> - 2019-09-02 13:37 +1000
          Re: Question re alt.comp.os.linux "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-09-02 19:54 -0400
            Re: Question re alt.comp.os.linux Peter 'Shaggy' Haywood <phaywood@alphalink.com.au> - 2019-09-04 19:37 +1000
      Re: Question re alt.comp.os.linux Peter 'Shaggy' Haywood <phaywood@alphalink.com.au> - 2019-09-02 13:34 +1000

csiph-web