Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #223256 > unrolled thread

How long will this take?

Back to article view | Back to linux.debian.user

Contents

  How long will this take? Matthew Campbell <trenix25@pm.me> - 2020-06-08 22:30 +0200
    Re: How long will this take? Dan Ritter <dsr@randomstring.org> - 2020-06-08 22:40 +0200
      Re: How long will this take? deloptes <deloptes@gmail.com> - 2020-06-08 23:00 +0200
        Re: How long will this take? Dan Ritter <dsr@randomstring.org> - 2020-06-09 00:10 +0200
          Re: How long will this take? Michael Stone <mstone@debian.org> - 2020-06-10 16:10 +0200
            Re: How long will this take? Michael Stone <mstone@debian.org> - 2020-06-10 17:00 +0200
            Re: How long will this take? Andrei POPESCU <andreimpopescu@gmail.com> - 2020-06-10 17:00 +0200
            Re: How long will this take? David Christensen <dpchrist@holgerdanske.com> - 2020-06-11 02:40 +0200
      Re: How long will this take? Jude DaShiell <jdashiel@panix.com> - 2020-06-09 02:20 +0200
        Re: How long will this take? Andrei POPESCU <andreimpopescu@gmail.com> - 2020-06-09 09:40 +0200
    Re: How long will this take? Nicolas George <george@nsup.org> - 2020-06-08 22:40 +0200
      Re: How long will this take? Matthew Campbell <trenix25@pm.me> - 2020-06-08 23:10 +0200
        Re: How long will this take? Nicolas George <george@nsup.org> - 2020-06-08 23:20 +0200
    Re: How long will this take? David Wright <deblis@lionunicorn.co.uk> - 2020-06-09 05:10 +0200
      Re: How long will this take? Christopher David Howie <me@chrishowie.com> - 2020-06-09 06:00 +0200
        Re: How long will this take? Nicolas George <george@nsup.org> - 2020-06-09 11:50 +0200
          Fw: How long will this take? Matthew Campbell <trenix25@pm.me> - 2020-06-09 17:40 +0200
          Re: How long will this take? Christopher David Howie <me@chrishowie.com> - 2020-06-10 02:50 +0200
      Re: How long will this take? Michael Stone <mstone@debian.org> - 2020-06-10 16:20 +0200
        Re: How long will this take? David Wright <deblis@lionunicorn.co.uk> - 2020-06-10 19:10 +0200
          Fw: How long will this take? Matthew Campbell <trenix25@pm.me> - 2020-06-10 19:20 +0200
            Re: Fw: How long will this take? David Christensen <dpchrist@holgerdanske.com> - 2020-06-11 03:10 +0200
              Fw: Fw: How long will this take? Matthew Campbell <trenix25@pm.me> - 2020-06-11 20:00 +0200
                Re: Fw: Fw: How long will this take? Dan Ritter <dsr@randomstring.org> - 2020-06-11 20:10 +0200
                  Re: Fw: Fw: How long will this take? Kenneth Parker <sea7kenp@gmail.com> - 2020-06-11 22:20 +0200
                  Re: Fw: Fw: How long will this take? "Rick Thomas" <rick.thomas@pobox.com> - 2020-06-13 11:20 +0200
          Re: How long will this take? Michael Stone <mstone@debian.org> - 2020-06-10 21:00 +0200
            [OT] How to help the OP? (was: How long will this take?) l0f4r0@tuta.io - 2020-06-10 21:30 +0200
              Re: [OT] How to help the OP? (was: How long will this take?) Nicolas George <george@nsup.org> - 2020-06-10 21:30 +0200
                Re: [OT] How to help the OP? (was: How long will this take?) Nicolas George <george@nsup.org> - 2020-06-10 22:00 +0200
                  Re: [OT] How to help the OP? (was: How long will this take?) Michael Stone <mstone@debian.org> - 2020-06-10 22:00 +0200
                    Re: [OT] How to help the OP? (was: How long will this take?) Nicolas George <george@nsup.org> - 2020-06-10 22:30 +0200
                    Re: [OT] How to help the OP? (was: How long will this take?) deloptes <deloptes@gmail.com> - 2020-06-11 07:30 +0200
                      Re: [OT] How to help the OP? (was: How long will this take?) <tomas@tuxteam.de> - 2020-06-11 07:50 +0200
                        Re: [OT] How to help the OP? (was: How long will this take?) deloptes <deloptes@gmail.com> - 2020-06-11 11:00 +0200
                          Re: [OT] How to help the OP? (was: How long will this take?) <tomas@tuxteam.de> - 2020-06-11 11:40 +0200
                          Re: [OT] How to help the OP? (was: How long will this take?) Michael Stone <mstone@debian.org> - 2020-06-11 14:00 +0200
                            Re: [OT] How to help the OP? (was: How long will this take?) Nicolas George <george@nsup.org> - 2020-06-11 14:10 +0200
                Re: [OT] How to help the OP? (was: How long will this take?) Michael Stone <mstone@debian.org> - 2020-06-10 22:00 +0200
              Re: [OT] How to help the OP? (was: How long will this take?) Michael Stone <mstone@debian.org> - 2020-06-10 21:50 +0200
                Re: [OT] How to help the OP? (was: How long will this take?) David Wright <deblis@lionunicorn.co.uk> - 2020-06-12 04:00 +0200
            Re: How long will this take? David Wright <deblis@lionunicorn.co.uk> - 2020-06-12 04:00 +0200
              Re: How long will this take? Michael Stone <mstone@debian.org> - 2020-06-12 14:00 +0200
                Re: How long will this take? David Wright <deblis@lionunicorn.co.uk> - 2020-06-19 04:20 +0200
                  Re: How long will this take? David Christensen <dpchrist@holgerdanske.com> - 2020-06-20 00:00 +0200
                    Re: How long will this take? David Wright <deblis@lionunicorn.co.uk> - 2020-06-26 15:10 +0200
                      Re: How long will this take? David Christensen <dpchrist@holgerdanske.com> - 2020-06-27 00:10 +0200
                        Re: How long will this take? David Wright <deblis@lionunicorn.co.uk> - 2020-06-27 03:30 +0200
                          Re: How long will this take? David Christensen <dpchrist@holgerdanske.com> - 2020-06-27 05:00 +0200
                            Re: How long will this take? David Wright <deblis@lionunicorn.co.uk> - 2020-07-02 03:10 +0200
    Re: How long will this take? David Christensen <dpchrist@holgerdanske.com> - 2020-06-09 09:10 +0200
    Re: How long will this take? l0f4r0@tuta.io - 2020-06-09 20:10 +0200
      Re: How long will this take? Nicolas George <george@nsup.org> - 2020-06-09 20:30 +0200
        Re: How long will this take? Jude DaShiell <jdashiel@panix.com> - 2020-06-09 21:50 +0200
          Re: How long will this take? Nicolas George <george@nsup.org> - 2020-06-09 21:50 +0200
        Re: How long will this take? Anders Andersson <pipatron@gmail.com> - 2020-06-10 12:50 +0200
          Re: How long will this take? Nicolas George <george@nsup.org> - 2020-06-10 13:20 +0200
            Re: How long will this take? Anders Andersson <pipatron@gmail.com> - 2020-06-10 15:30 +0200
              Re: How long will this take? Nicolas George <george@nsup.org> - 2020-06-10 15:40 +0200
                Re: How long will this take? Anders Andersson <pipatron@gmail.com> - 2020-06-10 16:00 +0200
                  Re: How long will this take? Nicolas George <george@nsup.org> - 2020-06-10 16:10 +0200
                    Re: How long will this take? Anders Andersson <pipatron@gmail.com> - 2020-06-10 22:10 +0200
                  Fw: How long will this take? Matthew Campbell <trenix25@pm.me> - 2020-06-10 19:00 +0200
      Re: How long will this take? Jude DaShiell <jdashiel@panix.com> - 2020-06-09 20:30 +0200
    Re: How long will this take? Seeds Notoneofmy <notoneofmyseeds@gmx.de> - 2020-06-27 01:40 +0200
      Re: How long will this take? rhkramer@gmail.com - 2020-06-27 03:30 +0200
        Re: How long will this take? Seeds Notoneofmy <notoneofmyseeds@gmx.de> - 2020-06-27 03:50 +0200
          Re: How long will this take? rhkramer@gmail.com - 2020-06-27 20:10 +0200
      Re: How long will this take? Andrei POPESCU <andreimpopescu@gmail.com> - 2020-06-27 10:40 +0200
      On Subject drift [was: How long will this take?] <tomas@tuxteam.de> - 2020-06-27 11:00 +0200
        Re: On Subject drift [was: How long will this take?] l0f4r0@tuta.io - 2020-06-27 11:40 +0200
        Re: On Subject drift [was: How long will this take?] Seeds Notoneofmy <notoneofmyseeds@gmx.de> - 2020-06-27 16:40 +0200
          Re: On Subject drift [was: How long will this take?] <tomas@tuxteam.de> - 2020-06-27 18:50 +0200

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

#223363 — Re: [OT] How to help the OP? (was: How long will this take?)

On Wed 10 Jun 2020 at 15:48:01 (-0400), Michael Stone wrote:
> On Wed, Jun 10, 2020 at 09:20:50PM +0200, l0f4r0@tuta.io wrote:
> > 10 juin 2020 à 20:51 de mstone@debian.org:
> > > On Wed, Jun 10, 2020 at 12:02:13PM -0500, David Wright wrote:
> > >
> > >> Both l0f4r0 and I have asked why the OP
> > >> is zeroing the drive, but no reply yet. Perhaps you can suggest an
> > >> answer.
> > >>
> > > I don't really care why the OP is doing it. I can think of several possibilities, but I don't see any need to argue with him over it. At some point it's reasonable to simply accept that someone is trying to do something and either help or not.

I haven't seen anyone arguing with the OP, even though there are
several arguments elsewhere in the thread.

> > It's the usual dilemma on mailing-lists/forums:
> > * Are the respondents supposed to answer the  OP question directly without hindsight (potential XY problem sometimes - http://xyproblem.info/)?
> > * Or are they expected to put the issue into perspective and challenge it? I know it can be frustrating because this is not always the answer we as OP want to read but it can be really instructive as we understand we were on a wrong track.
> > 
> > I'm still divided on this subject, maybe we need both depending on the context/OP...

I don't see it as a dilemma; it's been opined many a time here that
people answer the question however they want. Sometimes that might
involve asking questions, and on occasions a thread might end up like
a round of Twenty Questions.

> In general, I think that once someone's been asked a question multiple
> times, especially when that question isn't really germane, it's just
> rude to keep insisting on an answer. JMO.

Who insisted on an answer? I didn't actually ask any direct question
of the OP, and l0f4r0 merely asked out of curiosity. You seem to
imply that the OP was being hectored. All my direct questions have
been asked of you. Thanks for the answers.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#223364

On Wed 10 Jun 2020 at 14:51:32 (-0400), Michael Stone wrote:
> On Wed, Jun 10, 2020 at 12:02:13PM -0500, David Wright wrote:

[snipped the first part as it's covered elsewhere]

> > My use case for badblocks was closer to that of the OP, but still
> > different. Firstly, the disk contained personal data from unencrypted
> > use in the past. Secondly, I was intending to use it encrypted (as
> > mentioned) and prefer no high-watermark.  Thirdly, because of its
> > age (2011), I was interested in seeing how well it performed. I have
> > no idea whether the disk is "modern" in the sense you used, as I don't
> > follow the technology like some people on this list evidently do.
> > Fourthly, I don't make a habit of throwing away 2TB disks.
> 
> badblocks isn't particularly useful for achieving any of those goals
> vs just writing zeros. "modern" in this context means anything since
> probably the mid 90s but my memory is a bit fuzzy on the exact dates.
> certainly anything since the turn of the century.
> 
> > But, as you know about these things, a few questions:
> > 
> > . How does badblocks do its job in readonly mode, given that it
> >  doesn't know what any block's content ought to be.
> 
> you have to write the test data ahead of time
> 
> > . Why might the OP run badblocks, particularly non-destructively
> >  (as if to preserve something), and *then* zero the drive.
> 
> the only person I saw mention badblocks in this thread was you, but I
> guess I might have missed it

No, you're right, I brought it up, and I *am* conflating two things:
the OP running an unspecified "read test", reading every sector
looking for errors, and a hypothetical person running badblocks.

If you were preserving the disk contents (imagine there were
proprietary encryption software on it), and performed a "read test"
or ran badblocks on it, would that be sufficient to test the disk's
performance, as it's merely reading the sectors. Or do you have to
actually write, with badblocks -r for example?

> > . What's the easiest way of finding out about "consistent bad
> >  (not remappable) sectors" on a drive, as I soon will have to
> >  repeat this result (if not by this exact process) with a 3TB
> >  disk of 2013 vintage. (The good news: it has a USB3 connection.)
> 
> you'll get a bunch of errors while writing, and probably the drive
> will drop offline. you can use smartctl in the smartmontools package
> to see the status of retries & remapped sectors and get a health
> report on the drive, which you can use to decide whether to keep the
> drive in service even if it is currently working. (as a drive ages it
> will often record an increasing number of correctable errors, which
> typically will result in failure in the not-distant future.)

OK, so as far as the 2TB disk is concerned, writing anything over the
entire disk will provoke the reporting and/or remapping of any bad
sectors by SMART, so you can then check the statistics.

The only unaddressed point in my use case is the prevention of a
high-water mark, because zeroing the drive achieves precisely the
opposite. What ought I to be running, instead of badblocks -w -t random,
to achieve that goal?

> a confounding factor is that you might also get write errors and
> dropped disk if there's a USB issue, separate from whether the drive
> is working properly. smartctl may help you understand whether there's
> a physical drive issue, and you can try different USB adapters, ports,
> and cables.

Actually, one of the difficulties I have with the 3TB disk is reading
its SMART information. The disk claims to collect and retain it, but
the ?protocol/?device interface (in the container?) prevents my
reading it successfully. But I'll ask about that in a separate post
sometime.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#223368

On Thu, Jun 11, 2020 at 08:52:10PM -0500, David Wright wrote:
>If you were preserving the disk contents (imagine there were
>proprietary encryption software on it), and performed a "read test"
>or ran badblocks on it, would that be sufficient to test the disk's
>performance, as it's merely reading the sectors. Or do you have to
>actually write, with badblocks -r for example?

That really depends on whether you want to test read or write 
performance. If you mean that you want to test for correct operation 
rather than performance then you need a write test to fully exercise the 
disk. Some errors will only be found on write, and conversely some read 
errors will be corrected (remapped) on write.

>The only unaddressed point in my use case is the prevention of a
>high-water mark, because zeroing the drive achieves precisely the
>opposite. What ought I to be running, instead of badblocks -w -t random,
>to achieve that goal?

Create the encrypted volume first, then write zeros to it. :)

[toc] | [prev] | [next] | [standalone]

#223641

On Fri 12 Jun 2020 at 07:51:30 (-0400), Michael Stone wrote:
> On Thu, Jun 11, 2020 at 08:52:10PM -0500, David Wright wrote:

> > The only unaddressed point in my use case is the prevention of a
> > high-water mark, because zeroing the drive achieves precisely the
> > opposite. What ought I to be running, instead of badblocks -w -t random,
> > to achieve that goal?
> 
> Create the encrypted volume first, then write zeros to it. :)

Duh! That should work a treat. My posting that example bore me fruit.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#223682

On 2020-06-18 19:13, David Wright wrote:
> On Fri 12 Jun 2020 at 07:51:30 (-0400), Michael Stone wrote:
>> On Thu, Jun 11, 2020 at 08:52:10PM -0500, David Wright wrote:
> 
>>> The only unaddressed point in my use case is the prevention of a
>>> high-water mark, because zeroing the drive achieves precisely the
>>> opposite. What ought I to be running, instead of badblocks -w -t random,
>>> to achieve that goal?
>>
>> Create the encrypted volume first, then write zeros to it. :)
> 
> Duh! That should work a treat. My posting that example bore me fruit.
> 
> Cheers,
> David.


Benchmark is one thing.  But, from a security viewpoint, writing zeros 
to an encrypted volume amounts to providing blocks of plaintext for 
corresponding blocks of cyphertext, thereby facilitating cryptanalysis.


David

[toc] | [prev] | [next] | [standalone]

#224038

On Fri 19 Jun 2020 at 14:52:11 (-0700), David Christensen wrote:
> On 2020-06-18 19:13, David Wright wrote:
> > On Fri 12 Jun 2020 at 07:51:30 (-0400), Michael Stone wrote:
> > > On Thu, Jun 11, 2020 at 08:52:10PM -0500, David Wright wrote:
> > 
> > > > The only unaddressed point in my use case is the prevention of a
> > > > high-water mark, because zeroing the drive achieves precisely the
> > > > opposite. What ought I to be running, instead of badblocks -w -t random,
> > > > to achieve that goal?
> > > 
> > > Create the encrypted volume first, then write zeros to it. :)
> > 
> > Duh! That should work a treat. My posting that example bore me fruit.
> 
> Benchmark is one thing.  But, from a security viewpoint, writing zeros
> to an encrypted volume amounts to providing blocks of plaintext for
> corresponding blocks of cyphertext, thereby facilitating
> cryptanalysis.

So in view of the unlikelihood of badblocks actually logging something
more useful than SMART (where available) or normal disk write errors,
perhaps a compromise (for my use case) is to just write /dev/urandom
rather than /dev/zero. On this slow machine with an oldish PATA disk,
I can get about 75% speed from urandom, 15MB/s vs 20MB/s on a 29GiB
partition (no encryption). There's a noticeable slowdown because,
I presume, the machine runs a bit short of entropy after a while.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#224068

On 2020-06-26 06:07, David Wright wrote:
> On Fri 19 Jun 2020 at 14:52:11 (-0700), David Christensen wrote:

>> Benchmark is one thing.  But, from a security viewpoint, writing zeros
>> to an encrypted volume amounts to providing blocks of plaintext for
>> corresponding blocks of cyphertext, thereby facilitating
>> cryptanalysis.
> 
> So in view of the unlikelihood of badblocks actually logging something
> more useful than SMART (where available) or normal disk write errors,
> perhaps a compromise (for my use case) is to just write /dev/urandom
> rather than /dev/zero. 

Copying random data to a partition while creating an encrypted 
filesystem provides a high-entropy backdrop to conceal ciphertext 
blocks.  This is a form of steganography.  The Debian Installer manual 
partitioning page has an option to do this.


As the storage is used, the initial random blocks will be overwritten by 
ciphertext blocks.  Depending upon filesystem, encryption, volume 
management, and/or device details, the steganography degrades and may 
eventually disappear.


Copying random data to storage will add fresh nearly-random blocks on 
the device, improving the steganography.  (The canonical example is to 
copy /dev/urandom to a file until the filesystem fills up, and then 
delete the file.  But, this takes time and adds wear to the device.)


> On this slow machine with an oldish PATA disk,
> I can get about 75% speed from urandom, 15MB/s vs 20MB/s on a 29GiB
> partition (no encryption). There's a noticeable slowdown because,
> I presume, the machine runs a bit short of entropy after a while.

I think you are noticing a slowdown when the Linux write buffer fills.


David

[toc] | [prev] | [next] | [standalone]

#224075

On Fri 26 Jun 2020 at 15:06:31 (-0700), David Christensen wrote:
> On 2020-06-26 06:07, David Wright wrote:
> > On Fri 19 Jun 2020 at 14:52:11 (-0700), David Christensen wrote:
> 
> > > Benchmark is one thing.  But, from a security viewpoint, writing zeros
> > > to an encrypted volume amounts to providing blocks of plaintext for
> > > corresponding blocks of cyphertext, thereby facilitating
> > > cryptanalysis.
> > 
> > So in view of the unlikelihood of badblocks actually logging something
> > more useful than SMART (where available) or normal disk write errors,
> > perhaps a compromise (for my use case) is to just write /dev/urandom
> > rather than /dev/zero.
> 
> Copying random data to a partition while creating an encrypted
> filesystem provides a high-entropy backdrop to conceal ciphertext
> blocks.  This is a form of steganography.  The Debian Installer manual
> partitioning page has an option to do this.

I presume you meet this option when you select "Configure encrypted volumes",
something that I've never done. Because currently I only encrypt /home
and swap, I set these up after installation, if they're not already there.

I must admit that I prefer to partition disks and set up encryption
outside the d-i, usually capturing the process with script.

> As the storage is used, the initial random blocks will be overwritten
> by ciphertext blocks.  Depending upon filesystem, encryption, volume
> management, and/or device details, the steganography degrades and may
> eventually disappear.
> 
> Copying random data to storage will add fresh nearly-random blocks on
> the device, improving the steganography.  (The canonical example is to
> copy /dev/urandom to a file until the filesystem fills up, and then
> delete the file.  But, this takes time and adds wear to the device.)

Yes, SSD caveat taken on board.

> > On this slow machine with an oldish PATA disk,
> > I can get about 75% speed from urandom, 15MB/s vs 20MB/s on a 29GiB
> > partition (no encryption). There's a noticeable slowdown because,
> > I presume, the machine runs a bit short of entropy after a while.
> 
> I think you are noticing a slowdown when the Linux write buffer fills.

I'm not sure where these write buffers might be hiding: the
2000-vintage PC has 512MB memory, and the same size swap partition,
though the latter is on a disk constructed one month earlier than the
target disk (Feb/Mar 2008). The target disk has 8MB of cache.
With a leisurely determination of dd's PID, my first USR1 poke
occurred no earlier than after 4GB of copying, over three minutes in.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#224081

On 2020-06-26 18:25, David Wright wrote:
> On Fri 26 Jun 2020 at 15:06:31 (-0700), David Christensen wrote:
>> On 2020-06-26 06:07, David Wright wrote:

>>> On this slow machine with an oldish PATA disk,
>>> I can get about 75% speed from urandom, 15MB/s vs 20MB/s on a 29GiB
>>> partition (no encryption). There's a noticeable slowdown because,
>>> I presume, the machine runs a bit short of entropy after a while.
>>
>> I think you are noticing a slowdown when the Linux write buffer fills.
> 
> I'm not sure where these write buffers might be hiding: the
> 2000-vintage PC has 512MB memory, and the same size swap partition,
> though the latter is on a disk constructed one month earlier than the
> target disk (Feb/Mar 2008). The target disk has 8MB of cache.
> With a leisurely determination of dd's PID, my first USR1 poke
> occurred no earlier than after 4GB of copying, over three minutes in.

I seem to recall that most of my EIDE interfaces and drives were 100 
MB/s.  (A few were 133 MB/s.)  So, bulk reads or writes can completely 
use an 8 MB cache in a fraction of a second.


top(1) reports memory statistics on line 4.  I believe "buff/cache" is 
the amount of memory being used for I/O write buffering and read 
caching.  Line 5 has statics for swap.  I do not know if memory write 
buffer / read cache usage interacts with swap usage, but it would not 
surprise me.  top(1) should be able to show you.


Perhaps I misinterpreted your "slowdown" statement.  I assumed you ran a 
command similar to:

# dd if=/dev/urandom of=/dev/sdxn bs=1M status=progress


dd(1) is copying PRN data from the CPU to the kernel write buffer (in 
memory) and the kernel input/ output stack is copying from the write 
buffer to the HDD (likely via direct memory access, DMA).  The 
'status=progress' option will cause dd(1) to display the rate at which 
the write buffer is being filled.  I am not sure how to monitor the rate 
at which the write buffer is being drained.  Assuming the write buffer 
is initially empty, the filling process is "fast", and the draining 
process is "slow" when the above command is started, dd(1) should show 
fast throughput until the write buffer fills and then show slow 
throughput for the remainder of the transfer.  And, without a 'sync' 
option to dd(1), dd(1) will exit and the shell will display the next 
prompt as the final write buffer contents are being written to the HDD 
(e.g. the HDD will be busy for a short while after dd(1) is finished).


Another possibility -- magnetic disk drives have more sectors in outer 
tracks (lower sector number) than they have in inner tracks (higher 
sector number).  When filling an entire drive, I have seen the transfer 
rate drop by 40~50% over the duration of the transfer.  This is normal. 
Is this what you are referring to?


David

[toc] | [prev] | [next] | [standalone]

#224275

On Fri 26 Jun 2020 at 19:50:21 (-0700), David Christensen wrote:
> On 2020-06-26 18:25, David Wright wrote:
> > On Fri 26 Jun 2020 at 15:06:31 (-0700), David Christensen wrote:
> > > On 2020-06-26 06:07, David Wright wrote:
> 
> > > > On this slow machine with an oldish PATA disk,
> > > > I can get about 75% speed from urandom, 15MB/s vs 20MB/s on a 29GiB
> > > > partition (no encryption). There's a noticeable slowdown because,
> > > > I presume, the machine runs a bit short of entropy after a while.
> > > 
> > > I think you are noticing a slowdown when the Linux write buffer fills.
> > 
> > I'm not sure where these write buffers might be hiding: the
> > 2000-vintage PC has 512MB memory, and the same size swap partition,
> > though the latter is on a disk constructed one month earlier than the
> > target disk (Feb/Mar 2008). The target disk has 8MB of cache.
> > With a leisurely determination of dd's PID, my first USR1 poke
> > occurred no earlier than after 4GB of copying, over three minutes in.
> 
> I seem to recall that most of my EIDE interfaces and drives were 100
> MB/s.  (A few were 133 MB/s.)  So, bulk reads or writes can completely
> use an 8 MB cache in a fraction of a second.

This is IDE. The buses run at 100MHz, but I don't know where the
bottlenecks are. The idea was only to compare writing zeros and
random data. The machine, a 650MHz SE440BX-2 (Seattle 2), was selected
on the basis that it's presently housing a secondary drive with two
spare "root filesystem partitions" (which was in the Dell Optiplex
that died last month). It was doing nothing but running two ssh
sessions, one for dd and one for kill.

> top(1) reports memory statistics on line 4.  I believe "buff/cache" is
> the amount of memory being used for I/O write buffering and read
> caching.  Line 5 has statics for swap.  I do not know if memory write
> buffer / read cache usage interacts with swap usage, but it would not
> surprise me.  top(1) should be able to show you.
> 
> Perhaps I misinterpreted your "slowdown" statement.  I assumed you ran
> a command similar to:
> 
> # dd if=/dev/urandom of=/dev/sdxn bs=1M status=progress

Close: I was running within a script command, so I just poked
the dd occasionally with   kill -USR1   to record its progress
in the typescript file.

> dd(1) is copying PRN data from the CPU to the kernel write buffer (in
> memory) and the kernel input/ output stack is copying from the write
> buffer to the HDD (likely via direct memory access, DMA).  The
> 'status=progress' option will cause dd(1) to display the rate at which
> the write buffer is being filled.  I am not sure how to monitor the
> rate at which the write buffer is being drained.  Assuming the write
> buffer is initially empty, the filling process is "fast", and the
> draining process is "slow" when the above command is started, dd(1)
> should show fast throughput until the write buffer fills and then show
> slow throughput for the remainder of the transfer.  And, without a
> 'sync' option to dd(1), dd(1) will exit and the shell will display the
> next prompt as the final write buffer contents are being written to
> the HDD (e.g. the HDD will be busy for a short while after dd(1) is
> finished).
> 
> Another possibility -- magnetic disk drives have more sectors in outer
> tracks (lower sector number) than they have in inner tracks (higher
> sector number).  When filling an entire drive, I have seen the
> transfer rate drop by 40~50% over the duration of the transfer.  This
> is normal. Is this what you are referring to?

I tried to take account of these possibilities by using 29GB
partitions, much larger than the buffer sizes, and writing
two different partitions. But replicating the run a few times
didn't give me consistent enough timings to have confidence in
any conclusions. When I tried using sync to reduce the effect of
buffering, things slowed so much that I suspect there would be
no shortage of entropy anyway.

Regardless, the loss of speed is not serious enough for me to change
my strategy from:
  urandom before cryptsetup,
  zero before encrypting swap,
  zero to erase disk at end of life/possession.
I *have* given up running badblocks.

(The disk layout is:

Device         Start       End   Sectors   Size Type
/dev/sdb1       2048      8191      6144     3M BIOS boot
/dev/sdb2       8192   1023999   1015808   496M EFI System
/dev/sdb3    1024000   2047999   1024000   500M Linux swap
/dev/sdb4    2048000  63487999  61440000  29.3G Linux filesystem
/dev/sdb5   63488000 124927999  61440000  29.3G Linux filesystem
/dev/sdb6  124928000 976773119 851845120 406.2G Linux filesystem
)

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#223273

On 2020-06-08 13:22, Matthew Campbell wrote:
> I bought a new 4 terrabyte hard drive that is connected with a USB cable using USB2. It took about 32 hours to read every sector on the drive to look for bad sectors. I started blanking the sectors using /dev/zero last Friday night. It still isn't done. Is there a way I can find out how much data a particular process has written to the disk? I'm using Debian 10.4.
> 
> dd if=/dev/zero of=/dev/sdb ibs=4096 count=976754646
> 
> name=Matthew%20Campbell&email=trenix25%40pm.me

Install 'nmon'.  Then start a terminal and run 'nmon'. Press 'd' to 
display the disk monitoring screen.  This will show read throughput, 
write throughput, and percent utilization.


Alternatively, if you are using the Xfce desktop, add a Disk Performance 
Monitor applet to the panel and configure it for the correct device node 
/dev/sdX:

	Device			/dev/sdX
	unchecked Label		sdX
	Update interval(s)	1.000
	Monitor			Busy time
	checked Combine Read/Write data

Then hover your mouse pointer over the applet and it will show you read, 
write, and total statistics for both throughput and for busy time.


USB 2.0 ports have a maximum write speed around 25 MB/s.  eSATA ports 
(version 1) get much closer to their theoretical maximum of 150 MB/s. 
USB 3.0 beats them both.  Of course, you must have a fast drive and a 
fast program.


When using dd(1) to write blocks to a raw drive, use a block size of 1M 
(e.g. 1 Mibabyte).  As others have stated, a small block size of 4K will 
significantly reduce throughput due to I/O overhead.


Also as others have stated, writing zeros to an SSD may wear it out 
prematurely (depends upon internals of SSD).  The best approach is to do 
a "secure erase".


Rather than wiping storage devices with GNU/Linux userland tools, your 
best bet is to use the manufacturer's diagnostic utility.  In the ideal 
case, the utility sends a command to the drive controller and everything 
gets done internally at maximum speed.  I prefer the bootable "Live" 
tools, if available.  Each manufacturer has their own toolkit.  Get the 
one for your drive brand.  For example, SeaTools Bootable:

https://www.seagate.com/support/downloads/seatools/


David

[toc] | [prev] | [next] | [standalone]

#223288

Hi,

8 juin 2020 à 22:22 de trenix25@pm.me:

> I bought a new 4 terrabyte hard drive that is connected with a USB cable using USB2.  It took about 32 hours to read every sector on the drive to look for bad sectors.  I started blanking the sectors using /dev/zero last Friday night.
>
Out of curiosity, what is the purpose to wipe a brand new HDD?
Wouldn't formatting (or GPT overwrite) be sufficient?

9 juin 2020 à 08:59 de dpchrist@holgerdanske.com:

> Also as others have stated, writing zeros to an SSD may wear it out prematurely (depends upon internals of SSD).  The best approach is to do a "secure erase".
>
It seems to be a hard drive here ;)
> Rather than wiping storage devices with GNU/Linux userland tools, your best bet is to use the manufacturer's diagnostic utility.  In the ideal case, the utility sends a command to the drive controller and everything gets done internally at maximum speed.  I prefer the bootable "Live" tools, if available.  Each manufacturer has their own toolkit.  Get the one for your drive brand.  For example, SeaTools Bootable:
>
> https://www.seagate.com/support/downloads/seatools/
>
Even more true for an SSD (and yet, I'm not sure we can say "secure" for sure as those utilities are generally proprietary so we cannot verify what they do exactly).

Best regards,
l0f4r0

[toc] | [prev] | [next] | [standalone]

#223289

[Multipart message — attachments visible in raw view] — view raw

Jude DaShiell (12020-06-09):
> High security operations do this routinely.  They properly don't trust
> parts are as labeled from manufacturers especially manufacturers that
> send any of their stuff or get any of their stuff from China.

There is no trust to have. The previous contents would be overwritten on
the first actual write of a file.

And if the filesystem reads a sector that has never been written, that's
a serious bug in the operating system.

> I'm thinking of the binary search method and am wondering if disk
> operations of all sorts could be speeded up using it rather than
> sequential searches.  Or is binary already used now?

To search what?

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#223292

To search disk drives.
On Tue, 9 Jun 2020, Nicolas George wrote:

> Date: Tue, 9 Jun 2020 14:27:46
> From: Nicolas George <george@nsup.org>
> Reply-To: debian-user@lists.debian.org
> To: Jude DaShiell <jdashiel@panix.com>
> Cc: l0f4r0@tuta.io, Debian User <debian-user@lists.debian.org>
> Subject: Re: How long will this take?
>
> Jude DaShiell (12020-06-09):
> > High security operations do this routinely.  They properly don't trust
> > parts are as labeled from manufacturers especially manufacturers that
> > send any of their stuff or get any of their stuff from China.
>
> There is no trust to have. The previous contents would be overwritten on
> the first actual write of a file.
>
> And if the filesystem reads a sector that has never been written, that's
> a serious bug in the operating system.
>
> > I'm thinking of the binary search method and am wondering if disk
> > operations of all sorts could be speeded up using it rather than
> > sequential searches.  Or is binary already used now?
>
> To search what?
>
> Regards,
>
>

-- 

[toc] | [prev] | [next] | [standalone]

#223293

[Multipart message — attachments visible in raw view] — view raw

Jude DaShiell (12020-06-09):
> To search disk drives.

Are you still talking about binary search? To search disk drives for
what? Binary search is for sorted data. There is nothing sorted on a
hard drive. And binary search is when random access is fast; on
mechanical hard drives, random access is much slower than sequential
access.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#223299

On Tue, Jun 9, 2020 at 8:28 PM Nicolas George <george@nsup.org> wrote:
>
> Jude DaShiell (12020-06-09):
> > High security operations do this routinely.  They properly don't trust
> > parts are as labeled from manufacturers especially manufacturers that
> > send any of their stuff or get any of their stuff from China.
>
> There is no trust to have. The previous contents would be overwritten on
> the first actual write of a file.
>
> And if the filesystem reads a sector that has never been written, that's
> a serious bug in the operating system.

Too bad if you end up in a routine police investigation and they find
child pornography when scanning the disks for deleted files.

"Must have been the previous owner" is a valid defense, but I'd rather
not end up having to use it.

[toc] | [prev] | [next] | [standalone]

#223300

[Multipart message — attachments visible in raw view] — view raw

Anders Andersson (12020-06-10):
> Too bad if you end up in a routine police investigation and they find
> child pornography when scanning the disks for deleted files.
> 
> "Must have been the previous owner" is a valid defense, but I'd rather
> not end up having to use it.

Ah, but maybe the previous owner had discovered a cheap cure for
covid-19 and big pharma had them silenced. You would be wiping the last
traces of their research!

Seriously, first we were talking about hard drives straight from the
factory in China, making the thread… industrial espionage, I suppose?
And now we are talking about child pornography found in an unrelated
seizure.

So, for that to be relevant, you would need that all the following
conditions to be met:

- the previous owner had child pornography on this disk;

- unencrypted;

- they gave it away their disk in a way that makes it reusable;

- without wiping it themselves;

- cops show up at your door and take the drive to examine it;

- they do it before regular use has wiped it.

That is a fine Drake equation you got here, but maybe not a rational
justification for spending days wiping a drive.

For any security measure, it is easy to find afterwards a far-fetched
scenario where it makes a difference. But that is how TV writers work,
not security. For security, we must first define the attack model, and
then search for defense. Otherwise we end up barricading the back door
while the key to the front door is still under the mat.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#223304

On Wed, Jun 10, 2020 at 1:14 PM Nicolas George <george@nsup.org> wrote:
>
> Anders Andersson (12020-06-10):
> > Too bad if you end up in a routine police investigation and they find
> > child pornography when scanning the disks for deleted files.
> >
> > "Must have been the previous owner" is a valid defense, but I'd rather
> > not end up having to use it.
>
> Ah, but maybe the previous owner had discovered a cheap cure for
> covid-19 and big pharma had them silenced. You would be wiping the last
> traces of their research!
>
> Seriously, first we were talking about hard drives straight from the
> factory in China, making the thread… industrial espionage, I suppose?
> And now we are talking about child pornography found in an unrelated
> seizure.
>
> So, for that to be relevant, you would need that all the following
> conditions to be met:
>
> - the previous owner had child pornography on this disk;
>
> - unencrypted;
>
> - they gave it away their disk in a way that makes it reusable;
>
> - without wiping it themselves;
>
> - cops show up at your door and take the drive to examine it;
>
> - they do it before regular use has wiped it.
>
> That is a fine Drake equation you got here, but maybe not a rational
> justification for spending days wiping a drive.
>
> For any security measure, it is easy to find afterwards a far-fetched
> scenario where it makes a difference. But that is how TV writers work,
> not security. For security, we must first define the attack model, and
> then search for defense. Otherwise we end up barricading the back door
> while the key to the front door is still under the mat.

Except wiping a disk is trivial. Just start the job and come back
later to a clean disk. It's not like you have to wipe it by hand. I do
it routinely before I put a disk to use that's going to be used for a
couple of years.

[toc] | [prev] | [next] | [standalone]

#223307

[Multipart message — attachments visible in raw view] — view raw

Anders Andersson (12020-06-10):
> Except wiping a disk is trivial. Just start the job and come back
> later to a clean disk. It's not like you have to wipe it by hand. I do
> it routinely before I put a disk to use that's going to be used for a
> couple of years.

There is no "except" about: define your threat model; if it requires
wiping, wipe. If it does not, wiping is just a waste of time, little or
lots, still a waste. And it is a waste of power too.

There are many things that are trivial to do with a hard drive and could
benefit security in far-fetched scenarios. Did you wipe the possible
traces of cocaine? Did you weight it to check it matches the specs? Did
you take pictures of all angles? All these and many others are trivial.
Why one but not the others?

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#223308

On Wed, Jun 10, 2020 at 3:33 PM Nicolas George <george@nsup.org> wrote:
>
> Anders Andersson (12020-06-10):
> > Except wiping a disk is trivial. Just start the job and come back
> > later to a clean disk. It's not like you have to wipe it by hand. I do
> > it routinely before I put a disk to use that's going to be used for a
> > couple of years.
>
> There is no "except" about: define your threat model; if it requires
> wiping, wipe. If it does not, wiping is just a waste of time, little or
> lots, still a waste. And it is a waste of power too.
>
> There are many things that are trivial to do with a hard drive and could
> benefit security in far-fetched scenarios. Did you wipe the possible
> traces of cocaine? Did you weight it to check it matches the specs? Did
> you take pictures of all angles? All these and many others are trivial.
> Why one but not the others?

Because the police raiding my house for dealing drugs is not a
realistic threat. Looking at my drives for running Tor could be.

[toc] | [prev] | [next] | [standalone]

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

Back to top | Article view | linux.debian.user

csiph-web