Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > alt.os.linux > #53749 > unrolled thread

usb3 sticks do not seem to work properly.

Back to article view | Back to alt.os.linux

Contents

  usb3 sticks do not seem to work properly. William Unruh <unruh@invalid.ca> - 2019-02-11 16:42 +0000
    Re: usb3 sticks do not seem to work properly. Mike Easter <MikeE@ster.invalid> - 2019-02-11 08:56 -0800
    Re: usb3 sticks do not seem to work properly. Richard Kettlewell <invalid@invalid.invalid> - 2019-02-11 18:50 +0000
      Re: usb3 sticks do not seem to work properly. William Unruh <unruh@invalid.ca> - 2019-02-11 22:45 +0000
        Re: usb3 sticks do not seem to work properly. Jasen Betts <jasen@xnet.co.nz> - 2019-02-12 01:59 +0000
        Re: usb3 sticks do not seem to work properly. Richard Kettlewell <invalid@invalid.invalid> - 2019-02-12 08:45 +0000
          Re: usb3 sticks do not seem to work properly. Paul <nospam@needed.invalid> - 2019-02-12 05:50 -0500
          Re: usb3 sticks do not seem to work properly. William Unruh <unruh@invalid.ca> - 2019-02-12 16:03 +0000
            Re: usb3 sticks do not seem to work properly. Bit Twister <BitTwister@mouse-potato.com> - 2019-02-12 10:20 -0600
              Re: usb3 sticks do not seem to work properly. Peter Pearson <pkpearson@nowhere.invalid> - 2019-02-12 17:55 +0000
            Re: usb3 sticks do not seem to work properly. "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-02-12 13:56 -0500
              Re: usb3 sticks do not seem to work properly. Richard Kettlewell <invalid@invalid.invalid> - 2019-02-12 19:18 +0000
                Re: usb3 sticks do not seem to work properly. "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-02-12 14:59 -0500
            Re: usb3 sticks do not seem to work properly. Richard Kettlewell <invalid@invalid.invalid> - 2019-02-12 19:15 +0000
    Re: usb3 sticks do not seem to work properly. Wes Newell <w.newell@tx.rr.com> - 2019-02-11 20:27 +0000
    Re: usb3 sticks do not seem to work properly. "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-02-11 15:32 -0500
      Re: usb3 sticks do not seem to work properly. faeychild <faeychild@afraid.org> - 2019-02-12 08:32 +1100
      Re: usb3 sticks do not seem to work properly. William Unruh <unruh@invalid.ca> - 2019-02-11 22:49 +0000
        Re: usb3 sticks do not seem to work properly. "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-02-11 21:49 -0500
          Re: usb3 sticks do not seem to work properly. Paul <nospam@needed.invalid> - 2019-02-11 22:15 -0500
      Re: usb3 sticks do not seem to work properly. Jasen Betts <jasen@xnet.co.nz> - 2019-02-12 02:03 +0000
        Re: usb3 sticks do not seem to work properly. "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-02-11 22:15 -0500
          Re: usb3 sticks do not seem to work properly. "Carlos E.R." <robin_listas@es.invalid> - 2019-02-12 05:22 +0100
    Re: usb3 sticks do not seem to work properly. "Carlos E. R." <robin_listas@es.invalid> - 2019-02-11 22:28 +0100
    Re: usb3 sticks do not seem to work properly. Paul <nospam@needed.invalid> - 2019-02-11 18:52 -0500
      Re: usb3 sticks do not seem to work properly. ant@zimage.comANT (Ant) - 2019-02-11 21:47 -0600
        Re: usb3 sticks do not seem to work properly. William Unruh <unruh@invalid.ca> - 2019-02-12 04:22 +0000
          Re: usb3 sticks do not seem to work properly. "Carlos E. R." <robin_listas@es.invalid> - 2019-02-12 21:40 +0100
        Re: usb3 sticks do not seem to work properly. "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-02-11 23:29 -0500
          Re: usb3 sticks do not seem to work properly. Doug Laidlaw <laidlaws@hotkey.net.au> - 2019-02-13 06:40 +1100
            Re: usb3 sticks do not seem to work properly. Doug Laidlaw <laidlaws@hotkey.net.au> - 2019-02-14 03:01 +1100
              Re: usb3 sticks do not seem to work properly. "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-02-13 23:24 -0500
    Re: usb3 sticks do not seem to work properly. Jasen Betts <jasen@xnet.co.nz> - 2019-02-12 01:50 +0000

Page 1 of 2  [1] 2  Next page →

#53749 — usb3 sticks do not seem to work properly.

I am trying to install Mageia onto usb sticks, both usb2 and usb3
sticks. 
I am installing by doing 
dd if=Mageia-7-cauldron-x86_64.iso of=/dev/sdb bs=1M status=progress
and both usb2 and usb3 take a long time to install. Both start of fast,
but finally settle down to 10MB/s writing, so the writing takes almost
an hour. I thought usb3 was supposed to be fast. This is the same on my
laptop  (which has usb3 ports and MGA6) and the new desktop( Ubuntu 18). 
The big problem is that when I want to check tha tthe writing has
worked, I do 
md5sum /dev/sda
and it takes about 1/2 hour to check the 4GB content of the stick, and
persistantly, whether I used usb2 or usb4 sticks, the check sum is NOT
the same as original file.  The sticks all have plenty of room ( they
are either 16 or 32GB sticks) Furthermore the checksum changes if I run
md5sum again on the same stick with nothing done to it.
I might suspect a bad usb stick, but this has now been
true for four different sticks, two usb3 and two usb2. I once got the
correct checksum, but after 9 tries that was the only one. This is a bit
absurd. What in the world could be going on here? Am I doing something
wrong?

Why is it taking so long to write to usb 3 sticks.
Why is the checksum persistantly wrong and changing on successive checks
on the same stick?

[toc] | [next] | [standalone]

#53751

William Unruh wrote:
> I am trying to install Mageia onto usb sticks, both usb2 and usb3
> sticks.
> I am installing by doing
> dd if=Mageia-7-cauldron-x86_64.iso of=/dev/sdb bs=1M status=progress
> and both usb2 and usb3 take a long time to install. Both start of fast,
> but finally settle down to 10MB/s writing, so the writing takes almost
> an hour. I thought usb3 was supposed to be fast. This is the same on my
> laptop  (which has usb3 ports and MGA6) and the new desktop( Ubuntu 18).
> The big problem is that when I want to check tha tthe writing has
> worked, I do
> md5sum /dev/sda
> and it takes about 1/2 hour to check the 4GB content of the stick, and
> persistantly, whether I used usb2 or usb4 sticks, the check sum is NOT
> the same as original file.  The sticks all have plenty of room ( they
> are either 16 or 32GB sticks) Furthermore the checksum changes if I run
> md5sum again on the same stick with nothing done to it.
> I might suspect a bad usb stick, but this has now been
> true for four different sticks, two usb3 and two usb2. I once got the
> correct checksum, but after 9 tries that was the only one. This is a bit
> absurd. What in the world could be going on here? Am I doing something
> wrong?
> 
> Why is it taking so long to write to usb 3 sticks.
> Why is the checksum persistantly wrong and changing on successive checks
> on the same stick?

I don't know the answers to your questions, but I'll share with you my 
experiences from trying to install to USB 2.

It was difficult.  Once Pascal Hambourg  helped me tweak it to get it to 
boot, I discovered the USB was full of errors.

After I fixed all of those errors with fsck, the USB was very slow to 
boot and run compared to a USB 2 rigged to boot live with persistence 
and run in ram.

I decided that the install to USB was not a good idea for my goals 
compared to live with persistence and ram run.  The tool I used to live 
with persistence run in ram was mkusb which uses dd under the hood.  The 
tool I used to install to USB was the Ubuntu Ubiquity installer which 
performed poorly for me to install to USB.  The distro in all the above 
cases was Ubuntu Mate 18.04.



-- 
Mike Easter

[toc] | [prev] | [next] | [standalone]

#53754

William Unruh <unruh@invalid.ca> writes:
> I am trying to install Mageia onto usb sticks, both usb2 and usb3
> sticks. 
> I am installing by doing 
> dd if=Mageia-7-cauldron-x86_64.iso of=/dev/sdb bs=1M status=progress
> and both usb2 and usb3 take a long time to install. Both start of fast,
> but finally settle down to 10MB/s writing, so the writing takes almost
> an hour. I thought usb3 was supposed to be fast. This is the same on my
> laptop  (which has usb3 ports and MGA6) and the new desktop( Ubuntu 18). 
> The big problem is that when I want to check tha tthe writing has
> worked, I do 
> md5sum /dev/sda
> and it takes about 1/2 hour to check the 4GB content of the stick, and
> persistantly, whether I used usb2 or usb4 sticks, the check sum is NOT
> the same as original file.  The sticks all have plenty of room ( they
> are either 16 or 32GB sticks)

If you write a 4GB file to a 16GB USB stick, and then hash the whole
16GB, you’re not comparing like with like; you should not expect the
hashes to match.

If you write to /dev/sdb but read back from /dev/sda you shouldn’t
expect a match either...

> Furthermore the checksum changes if I run md5sum again on the same
> stick with nothing done to it.

That’s a more serious problem. You could:

1) Check the kernel log for error output
2) Do a thorough test on the USB stick. (I would use vbig[1], but the
   choice of tool is up to you.)

[1] https://github.com/ewxrjk/vbig/

> I might suspect a bad usb stick, but this has now been
> true for four different sticks, two usb3 and two usb2. I once got the
> correct checksum, but after 9 tries that was the only one. This is a bit
> absurd. What in the world could be going on here? Am I doing something
> wrong?
>
> Why is it taking so long to write to usb 3 sticks.
> Why is the checksum persistantly wrong and changing on successive checks
> on the same stick?
>
>

-- 
https://www.greenend.org.uk/rjk/

[toc] | [prev] | [next] | [standalone]

#53760

On 2019-02-11, Richard Kettlewell <invalid@invalid.invalid> wrote:
> William Unruh <unruh@invalid.ca> writes:
>> I am trying to install Mageia onto usb sticks, both usb2 and usb3
>> sticks. 
>> I am installing by doing 
>> dd if=Mageia-7-cauldron-x86_64.iso of=/dev/sdb bs=1M status=progress
>> and both usb2 and usb3 take a long time to install. Both start of fast,
>> but finally settle down to 10MB/s writing, so the writing takes almost
>> an hour. I thought usb3 was supposed to be fast. This is the same on my
>> laptop  (which has usb3 ports and MGA6) and the new desktop( Ubuntu 18). 
>> The big problem is that when I want to check tha tthe writing has
>> worked, I do 
>> md5sum /dev/sda
>> and it takes about 1/2 hour to check the 4GB content of the stick, and
>> persistantly, whether I used usb2 or usb4 sticks, the check sum is NOT
>> the same as original file.  The sticks all have plenty of room ( they
>> are either 16 or 32GB sticks)
>
> If you write a 4GB file to a 16GB USB stick, and then hash the whole
> 16GB, you’re not comparing like with like; you should not expect the
> hashes to match.

OK, that makes sense. I thought that the dd transfer simply cut off the
stick memory at the memory size of the file I was dd to it. 
Is there some way of testing whether the write to the stick worked?

>
> If you write to /dev/sdb but read back from /dev/sda you shouldn’t
> expect a match either...

Sorry. typo.

>
>> Furthermore the checksum changes if I run md5sum again on the same
>> stick with nothing done to it.

That happened once. I guess I should throw away that stick.

>
> That’s a more serious problem. You could:
>
> 1) Check the kernel log for error output
> 2) Do a thorough test on the USB stick. (I would use vbig[1], but the
>    choice of tool is up to you.)

They are cheap enough that it is not worth it. 

>
> [1] https://github.com/ewxrjk/vbig/
>
>> I might suspect a bad usb stick, but this has now been
>> true for four different sticks, two usb3 and two usb2. I once got the

The "check again and get a different checksum" only happened for one
stick. It was the "checksum of the stick was not the same as the file"
that happened again and again. You have explained that to me. 
I suspected that that was happening when I tried to transfer back the
file with dd if=/dev/sdb of=M.iso and I got a file size way larger than
what I ha transfered to the usb stick. 


>> correct checksum, but after 9 tries that was the only one. This is a bit
>> absurd. What in the world could be going on here? Am I doing something
>> wrong?
>>
>> Why is it taking so long to write to usb 3 sticks.
>> Why is the checksum persistantly wrong and changing on successive checks
>> on the same stick?

I suspect that the usb sockets on the front of the new computer are just
2.0, although looking at the motherboard this should not be true.
Plugging the stick into the back has it being read at 150MB/s which sure
looks more like usb3.
>>
>>
>

[toc] | [prev] | [next] | [standalone]

#53767

On 2019-02-11, William Unruh <unruh@invalid.ca> wrote:
>> If you write a 4GB file to a 16GB USB stick, and then hash the whole
>> 16GB, you’re not comparing like with like; you should not expect the
>> hashes to match.
>
> OK, that makes sense. I thought that the dd transfer simply cut off the
> stick memory at the memory size of the file I was dd to it. 
> Is there some way of testing whether the write to the stick worked?

read back the same size as the original

  dd if=/dev/something bs=something count=something | md5sum
  
>>> correct checksum, but after 9 tries that was the only one. This is a bit
>>> absurd. What in the world could be going on here? Am I doing something
>>> wrong?
>>>
>>> Why is it taking so long to write to usb 3 sticks.
>>> Why is the checksum persistantly wrong and changing on successive checks
>>> on the same stick?
>
> I suspect that the usb sockets on the front of the new computer are just
> 2.0, although looking at the motherboard this should not be true.
> Plugging the stick into the back has it being read at 150MB/s which sure
> looks more like usb3.

front usb3 sockets have a 20-pin plug for each pair (instead of 9-pin
for usb2)


-- 
  When I tried casting out nines I made a hash of it.

[toc] | [prev] | [next] | [standalone]

#53776

William Unruh <unruh@invalid.ca> writes:
> Richard Kettlewell <invalid@invalid.invalid> wrote:
>> William Unruh <unruh@invalid.ca> writes:
>>> I am trying to install Mageia onto usb sticks, both usb2 and usb3
>>> sticks. 
>>> I am installing by doing 
>>> dd if=Mageia-7-cauldron-x86_64.iso of=/dev/sdb bs=1M status=progress
>>> and both usb2 and usb3 take a long time to install. Both start of fast,
>>> but finally settle down to 10MB/s writing, so the writing takes almost
>>> an hour. I thought usb3 was supposed to be fast. This is the same on my
>>> laptop  (which has usb3 ports and MGA6) and the new desktop( Ubuntu 18). 
>>> The big problem is that when I want to check tha tthe writing has
>>> worked, I do 
>>> md5sum /dev/sda
>>> and it takes about 1/2 hour to check the 4GB content of the stick, and
>>> persistantly, whether I used usb2 or usb4 sticks, the check sum is NOT
>>> the same as original file.  The sticks all have plenty of room ( they
>>> are either 16 or 32GB sticks)
>>
>> If you write a 4GB file to a 16GB USB stick, and then hash the whole
>> 16GB, you’re not comparing like with like; you should not expect the
>> hashes to match.
>
> OK, that makes sense. I thought that the dd transfer simply cut off the
> stick memory at the memory size of the file I was dd to it. 

There’s no way to do this. The size of the storage is fixed.

> Is there some way of testing whether the write to the stick worked?

Read back the same size you wrote to it (e.g. using head -c) and hash
that.

In general I would move on from md5sum to at least sha256, but it
shouldn’t matter much for this particular application.

-- 
https://www.greenend.org.uk/rjk/

[toc] | [prev] | [next] | [standalone]

#53777

Richard Kettlewell wrote:
> William Unruh <unruh@invalid.ca> writes:
>> Richard Kettlewell <invalid@invalid.invalid> wrote:
>>> William Unruh <unruh@invalid.ca> writes:
>>>> I am trying to install Mageia onto usb sticks, both usb2 and usb3
>>>> sticks. 
>>>> I am installing by doing 
>>>> dd if=Mageia-7-cauldron-x86_64.iso of=/dev/sdb bs=1M status=progress
>>>> and both usb2 and usb3 take a long time to install. Both start of fast,
>>>> but finally settle down to 10MB/s writing, so the writing takes almost
>>>> an hour. I thought usb3 was supposed to be fast. This is the same on my
>>>> laptop  (which has usb3 ports and MGA6) and the new desktop( Ubuntu 18). 
>>>> The big problem is that when I want to check tha tthe writing has
>>>> worked, I do 
>>>> md5sum /dev/sda
>>>> and it takes about 1/2 hour to check the 4GB content of the stick, and
>>>> persistantly, whether I used usb2 or usb4 sticks, the check sum is NOT
>>>> the same as original file.  The sticks all have plenty of room ( they
>>>> are either 16 or 32GB sticks)
>>> If you write a 4GB file to a 16GB USB stick, and then hash the whole
>>> 16GB, you’re not comparing like with like; you should not expect the
>>> hashes to match.
>> OK, that makes sense. I thought that the dd transfer simply cut off the
>> stick memory at the memory size of the file I was dd to it. 
> 
> There’s no way to do this. The size of the storage is fixed.

The "Mageia-7-cauldron-x86_64.iso" we know to be a self contained,
consistent, collection of partitions. A "disc layout" of sorts.
If we transfer it completely, we can be assured that parsing the
MBR for the disc will not cause any structures to fall outside
the written area. In that sense, we don't have to worry the
"stick is too big". The stuff at the end is just "ignored".

If we do this

    sudo dd if=Mageia-7-cauldron-x86_64.iso of=/dev/sdb bs=1M

and the ISO was 1234MB long, then this will happen.

    +----------------------------------------------------+  16GB stick
    |                                                    |
    +----------------------------------------------------+

    +-------+-------+----------+-------------------------+  16GB stick
    | part1 | part2 |  part3   |      not touched        |
    +-------+-------+----------+-------------------------+
    <---- 1234MB ------------->

When I want to verify the writes that were done, I would

    md5sum Mageia-7-cauldron-x86_64.iso               # generate a hash with low collision
                                                      # probability. MD5 is good enough.

    sudo dd if=/dev/sdb of=- bs=1M count=1234 |  md5sum    # exactly copy the written part,
                                                           # send to STDOUT for pipe or...

    sudo dd if=/dev/sdb      bs=1M count=1234 |  md5sum    # STDOUT is the default "of="

The "dd" program doesn't look like much, but it's
actually a Swiss Army knife if in the hands of a villain :-)

It's got "seek" and "skip", which we haven't touched yet.

And if you need a program with a slightly different
writing discipline, there's always (some flavor of)
ddrescue.

    Paul

[toc] | [prev] | [next] | [standalone]

#53781

On 2019-02-12, Richard Kettlewell <invalid@invalid.invalid> wrote:
> William Unruh <unruh@invalid.ca> writes:
>> Richard Kettlewell <invalid@invalid.invalid> wrote:
>>> William Unruh <unruh@invalid.ca> writes:
>>>> I am trying to install Mageia onto usb sticks, both usb2 and usb3
>>>> sticks. 
>>>> I am installing by doing 
>>>> dd if=Mageia-7-cauldron-x86_64.iso of=/dev/sdb bs=1M status=progress
>>>> and both usb2 and usb3 take a long time to install. Both start of fast,
>>>> but finally settle down to 10MB/s writing, so the writing takes almost
>>>> an hour. I thought usb3 was supposed to be fast. This is the same on my
>>>> laptop  (which has usb3 ports and MGA6) and the new desktop( Ubuntu 18). 
>>>> The big problem is that when I want to check tha tthe writing has
>>>> worked, I do 
>>>> md5sum /dev/sda
>>>> and it takes about 1/2 hour to check the 4GB content of the stick, and
>>>> persistantly, whether I used usb2 or usb4 sticks, the check sum is NOT
>>>> the same as original file.  The sticks all have plenty of room ( they
>>>> are either 16 or 32GB sticks)
>>>
>>> If you write a 4GB file to a 16GB USB stick, and then hash the whole
>>> 16GB, you’re not comparing like with like; you should not expect the
>>> hashes to match.
>>
>> OK, that makes sense. I thought that the dd transfer simply cut off the
>> stick memory at the memory size of the file I was dd to it. 
>
> There’s no way to do this. The size of the storage is fixed.
>
>> Is there some way of testing whether the write to the stick worked?
>
> Read back the same size you wrote to it (e.g. using head -c) and hash
> that.
>
> In general I would move on from md5sum to at least sha256, but it
> shouldn’t matter much for this particular application.

Why? Is sha faster? AFAIK for detecting changes in a file (rather than
standing as a guardian against forgeries) md5 and any sha are equally
good. 

>

[toc] | [prev] | [next] | [standalone]

#53782

["Followup-To:" header set to alt.os.linux.mageia.]
On Tue, 12 Feb 2019 16:03:20 -0000 (UTC), William Unruh wrote:
> On 2019-02-12, Richard Kettlewell <invalid@invalid.invalid> wrote:

>>
>> In general I would move on from md5sum to at least sha256, but it
>> shouldn’t matter much for this particular application.
>
> Why? Is sha faster?

No.

> AFAIK for detecting changes in a file (rather than
> standing as a guardian against forgeries) md5 and any sha are equally
> good.

Not so. Seen articles years ago where criminals had a program to
append their malware then add the correct number/content bytes to the
file to cause collision of the md5sum to come out correct.


https://en.wikipedia.org/wiki/MD5

[toc] | [prev] | [next] | [standalone]

#53785

On Tue, 12 Feb 2019 10:20:29 -0600, Bit Twister wrote:
> ["Followup-To:" header set to alt.os.linux.mageia.]
> On Tue, 12 Feb 2019 16:03:20 -0000 (UTC), William Unruh wrote:
>> On 2019-02-12, Richard Kettlewell <invalid@invalid.invalid> wrote:
>
>>>
>>> In general I would move on from md5sum to at least sha256, but it
>>> shouldn’t matter much for this particular application.
>>
>> Why? Is sha faster?
>
> No.
>
>> AFAIK for detecting changes in a file (rather than
>> standing as a guardian against forgeries) md5 and any sha are equally
>> good.
>
> Not so. Seen articles years ago where criminals had a program to
> append their malware then add the correct number/content bytes to the
> file to cause collision of the md5sum to come out correct.
>
> https://en.wikipedia.org/wiki/MD5


William Unruh is presumably interested in the probability that a random
error would go undetected, and not concerned about the possibility
that the powers of darkness are interfering in his flash-drive
initialization.  For that purpose, MD5, SHA-1, and SHA-256 are all more
than adequate.

It seems that SHA-1 (for which collisions have also been found, but with
much more difficulty) is about 25% slower than MD5, while SHA-256 takes
a bit over 3 times longer (I hope I'm doing this right):

$ head -c 1G </dev/urandom >temp.tmp

$ time md5sum <temp.tmp
2de65a591115f6c70cfa9350aff4ab5c  -

real	0m1.642s
user	0m1.360s
sys	0m0.280s

$ time sha1sum <temp.tmp
0d32c484eb431f752053ff5615f506478109e238  -

real	0m1.963s
user	0m1.748s
sys	0m0.212s

$ time sha256sum <temp.tmp
f6aaf16f8da766f0e3c270525d9918376c443a3765f697764059c283d163031e  -

real	0m4.898s
user	0m4.728s
sys	0m0.168s


-- 
To email me, substitute nowhere->runbox, invalid->com.

[toc] | [prev] | [next] | [standalone]

#53788

On Tue, 12 Feb 2019 11:03:20 -0500, William Unruh <unruh@invalid.ca> wrote:

> Why? Is sha faster? AFAIK for detecting changes in a file (rather than
> standing as a guardian against forgeries) md5 and any sha are equally
> good.

While md5 is probably ok for this purpose, it's known to have possible
collisions where specific types of corruption could still result in
the same sum. It's extremely unlikely, but still considered too much
of a risk for md5 sums to be used for security purposes any more.
See https://en.wikipedia.org/wiki/Collision_attack

I prefer to check both md5 and sha512 sums. The likely hood of any
corruption, intentional or not, to produce valid collision attacks
against both is much, much lower.

Regards, Dave Hodgins

-- 
Change dwhodgins@nomail.afraid.org to davidwhodgins@teksavvy.com for
email replies.

[toc] | [prev] | [next] | [standalone]

#53790

"David W. Hodgins" <dwhodgins@nomail.afraid.org> writes:
> I prefer to check both md5 and sha512 sums. The likely hood of any
> corruption, intentional or not, to produce valid collision attacks
> against both is much, much lower.

Within the Merkle–Damgård, family (including both MD5 and SHA512) it’s
not as much lower as you think - see
https://www.iacr.org/archive/crypto2004/31520306/multicollisions.pdf

-- 
https://www.greenend.org.uk/rjk/

[toc] | [prev] | [next] | [standalone]

#53795

On Tue, 12 Feb 2019 14:18:45 -0500, Richard Kettlewell <invalid@invalid.invalid> wrote:

> "David W. Hodgins" <dwhodgins@nomail.afraid.org> writes:
>> I prefer to check both md5 and sha512 sums. The likely hood of any
>> corruption, intentional or not, to produce valid collision attacks
>> against both is much, much lower.
>
> Within the Merkle–Damgård, family (including both MD5 and SHA512) it’s
> not as much lower as you think - see
> https://www.iacr.org/archive/crypto2004/31520306/multicollisions.pdf

Thanks for the info, though in this case if my skimming is correct, it
is talking about using the same hash algorithm 2 or more times with
different seed values, not using two different algorithms to reduce
the possibility of valid collisions.

I'll read through it more carefully later.^M
^M
Regards, Dave Hodgins

-- 
Change dwhodgins@nomail.afraid.org to davidwhodgins@teksavvy.com for
email replies.

[toc] | [prev] | [next] | [standalone]

#53789

William Unruh <unruh@invalid.ca> writes:
> On 2019-02-12, Richard Kettlewell <invalid@invalid.invalid> wrote:
>> In general I would move on from md5sum to at least sha256, but it
>> shouldn’t matter much for this particular application.
>
> Why? Is sha faster?

No, less broken.

> AFAIK for detecting changes in a file (rather than standing as a
> guardian against forgeries) md5 and any sha are equally good.

As I said.

But as a general principle: once a primitive is broken in some way,
retiring it means you avoid the risk of mis-analyzing whether you’re at
risk from it.

That doesn’t mean retiring primitives is easy; SHA1 at least is going to
remain in some quite interesting places for a long time, for
hard-to-solve practical reasons.

-- 
https://www.greenend.org.uk/rjk/

[toc] | [prev] | [next] | [standalone]

#53756

On Mon, 11 Feb 2019 16:42:33 +0000, William Unruh wrote:

> Why is it taking so long to write to usb 3 sticks.
> Why is the checksum persistantly wrong and changing on successive checks
> on the same stick?

Forget md5sums as long as you know the iso file checks. Just 
dd if=Mageia-7-cauldron-x86_64.iso of=/dev/sdb and then boot it. I've done 
this at least 100 times with iso files, no problem. An hour sounds about 
right for a 4GB iso. It starts fast and slows down when the buffer gets 
full. Yes I'm also very disappointed in the speed of usb3 flash drives. I 
get about the same speed with plugging a usb3 drive into a usb2 port. 
Starts fast as heck then slows down. But they benchmark real fast.:-)

64GB usb3 flash drive. First in usb 3 port. Second in usb2 port.

root@mythfe0:/home/wes# hdparm -Tt /dev/sdc

/dev/sdc:
 Timing cached reads:   21402 MB in  2.00 seconds = 10716.67 MB/sec
 Timing buffered disk reads: 110 MB in  3.04 seconds =  36.14 MB/sec
root@mythfe0:/home/wes# hdparm -Tt /dev/sdc

/dev/sdc:
 Timing cached reads:   20738 MB in  2.00 seconds = 10383.03 MB/sec
 Timing buffered disk reads: 110 MB in  3.01 seconds =  36.58 MB/sec
root@mythfe0:/home/wes# 

[toc] | [prev] | [next] | [standalone]

#53757

On Mon, 11 Feb 2019 11:42:33 -0500, William Unruh <unruh@invalid.ca> wrote:

> I am trying to install Mageia onto usb sticks, both usb2 and usb3
> sticks.
> I am installing by doing
> dd if=Mageia-7-cauldron-x86_64.iso of=/dev/sdb bs=1M status=progress
> and both usb2 and usb3 take a long time to install. Both start of fast,
> but finally settle down to 10MB/s writing, so the writing takes almost
> an hour. I thought usb3 was supposed to be fast. This is the same on my
> laptop  (which has usb3 ports and MGA6) and the new desktop( Ubuntu 18).

While the usb interface can handle higher speeds, for example with an ssd
drive in a usb connected external drive, the sticks cannot. Most usb sticks
are low quality slow sticks. Faster ones are more expensive, and hard to
find as the cheap low quality ones have flooded the market.

At the start of writing, the in ram buffers are being filled up. Once those
are full the speed slows down to the actual physical write speed the stick
can handle.

One thing I found that helped, after I'd noticed that the sticks get very
warm after large amounts of writes, is to stick it in a freezer for a while
before using it. Seems to help.

As for checking the results, I usually copy the iso back onto a hard drive
being careful to ensure the resulting file is the same size as the original,
and then run the md5 and sha checksum commands on that copy.

While the dd command may (or may not) appear to end before the iso has
been fully copied, run htop, sort the output by the S (status) column,
and wait for the kernel processes to stop showing any device waits.
Until then, it's not fully written, so rebooting or removing the stick
will have partial results only.

Regards, Dave Hodgins

-- 
Change dwhodgins@nomail.afraid.org to davidwhodgins@teksavvy.com for
email replies.

[toc] | [prev] | [next] | [standalone]

#53759

On 2/12/19 7:32 AM, David W. Hodgins wrote:

> While the dd command may (or may not) appear to end before the iso has
> been fully copied, run htop, sort the output by the S (status) column,
> and wait for the kernel processes to stop showing any device waits.
> Until then, it's not fully written, so rebooting or removing the stick
> will have partial results only.
> 
> Regards, Dave Hodgins
> 

I fully agree with you, David.
I experienced the same thing with video transfers, even to USB3.

I also got annoyed fumbling about under the desk locating the USB socket 
on the box and now use a USB3  extension cable.

I still had buffering slowdowns until I tried a USB3 card reader 
(Transcend)  and SD cards - class 3.

The writes are now blazing fast.  So I don't think much of sticks except 
for archiving.

regards


-- 
faeychild
Running plasmashell 5.12.2 on 4.14.89-desktop-1.mga6 kernel.
Mageia release 6 (Official) for x86_64 installed via Mageia-6-x86_64-DVD.iso

[toc] | [prev] | [next] | [standalone]

#53761

On 2019-02-11, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote:
> On Mon, 11 Feb 2019 11:42:33 -0500, William Unruh <unruh@invalid.ca> wrote:
>
>> I am trying to install Mageia onto usb sticks, both usb2 and usb3
>> sticks.
>> I am installing by doing
>> dd if=Mageia-7-cauldron-x86_64.iso of=/dev/sdb bs=1M status=progress
>> and both usb2 and usb3 take a long time to install. Both start of fast,
>> but finally settle down to 10MB/s writing, so the writing takes almost
>> an hour. I thought usb3 was supposed to be fast. This is the same on my
>> laptop  (which has usb3 ports and MGA6) and the new desktop( Ubuntu 18).
>
> While the usb interface can handle higher speeds, for example with an ssd
> drive in a usb connected external drive, the sticks cannot. Most usb sticks
> are low quality slow sticks. Faster ones are more expensive, and hard to
> find as the cheap low quality ones have flooded the market.
>
> At the start of writing, the in ram buffers are being filled up. Once those
> are full the speed slows down to the actual physical write speed the stick
> can handle.
>
> One thing I found that helped, after I'd noticed that the sticks get very
> warm after large amounts of writes, is to stick it in a freezer for a while
> before using it. Seems to help.
>
> As for checking the results, I usually copy the iso back onto a hard drive
> being careful to ensure the resulting file is the same size as the original,

How do you copy the iso back. I put an Mageis is directly onto the
stick. Ie, there is not file on the stick. 


By the way, is it possible to rsync onto the stick instead of using dd?
That way the write is hash checked during the writing.


> and then run the md5 and sha checksum commands on that copy.
>
> While the dd command may (or may not) appear to end before the iso has
> been fully copied, run htop, sort the output by the S (status) column,
> and wait for the kernel processes to stop showing any device waits.
> Until then, it's not fully written, so rebooting or removing the stick
> will have partial results only.

Thanks. Maybe that is also what happened.
>
> Regards, Dave Hodgins
>

[toc] | [prev] | [next] | [standalone]

#53769

On Mon, 11 Feb 2019 17:49:17 -0500, William Unruh <unruh@invalid.ca> wrote:

> How do you copy the iso back. I put an Mageis is directly onto the
> stick. Ie, there is not file on the stick.

The reverse of the way it was copied to the stick, but with the count
parameter added. For example

dd if=/dev/sdu of=/root/copyfromstick.iso bs=1M count=2357

The tricky part is getting the blocksize and count to fit evently into the
size of the iso.

> By the way, is it possible to rsync onto the stick instead of using dd?
> That way the write is hash checked during the writing.

No. The rsync works by copying files within or between filesystems. The
dd command works with devices. While a partition within a device is also
considered a device, if a partition is specified, it's the entire partition
that is copied, including unused blocks, not a file by file copy.

Regards, Dave Hodgins

-- 
Change dwhodgins@nomail.afraid.org to davidwhodgins@teksavvy.com for
email replies.

[toc] | [prev] | [next] | [standalone]

#53771

David W. Hodgins wrote:
> On Mon, 11 Feb 2019 17:49:17 -0500, William Unruh <unruh@invalid.ca> wrote:
> 
>> How do you copy the iso back. I put an Mageis is directly onto the
>> stick. Ie, there is not file on the stick.
> 
> The reverse of the way it was copied to the stick, but with the count
> parameter added. For example
> 
> dd if=/dev/sdu of=/root/copyfromstick.iso bs=1M count=2357
> 
> The tricky part is getting the blocksize and count to fit evently into the
> size of the iso.

factor is a built-in on your OS.

factor 2471493632

2471493632: 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2357

= 1048576 * 2357
   bs=1M     count=2357

    Paul

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | alt.os.linux

csiph-web