Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #235221 > unrolled thread

OT: minimum bs for dd?

Back to article view | Back to linux.debian.user

Contents

  OT: minimum bs for dd? Bob Bernstein <poobah@ruptured-duck.com> - 2021-05-15 22:30 +0200
    Re: OT: minimum bs for dd? The Wanderer <wanderer@fastmail.fm> - 2021-05-15 22:40 +0200
    Re: OT: minimum bs for dd? IL Ka <kazakevichilya@gmail.com> - 2021-05-15 22:40 +0200
      Re: OT: minimum bs for dd? Bob Bernstein <bob@fanatick.org> - 2021-05-16 09:40 +0200
      Re: OT: minimum bs for dd? Michael Stone <mstone@debian.org> - 2021-05-16 19:50 +0200
        Re: OT: minimum bs for dd? Richard Owlett <rowlett@cloud85.net> - 2021-05-16 20:40 +0200
          Re: OT: minimum bs for dd? Stefan Monnier <monnier@iro.umontreal.ca> - 2021-05-17 00:40 +0200
            Re: OT: minimum bs for dd? The Wanderer <wanderer@fastmail.fm> - 2021-05-17 00:50 +0200
              Re: OT: minimum bs for dd? "Thomas Schmitt" <scdbackup@gmx.net> - 2021-05-17 09:00 +0200
                Re: OT: minimum bs for dd? The Wanderer <wanderer@fastmail.fm> - 2021-05-17 12:30 +0200
              Re: OT: minimum bs for dd? Greg Wooledge <greg@wooledge.org> - 2021-05-17 13:20 +0200
            Re: OT: minimum bs for dd? Andrei POPESCU <andreimpopescu@gmail.com> - 2021-05-17 08:10 +0200
              Re: OT: minimum bs for dd? <tomas@tuxteam.de> - 2021-05-17 08:40 +0200
                Re: OT: minimum bs for dd? Andrei POPESCU <andreimpopescu@gmail.com> - 2021-05-17 09:30 +0200
                  Re: OT: minimum bs for dd? David <bouncingcats@gmail.com> - 2021-05-17 10:00 +0200
                    Re: OT: minimum bs for dd? Andrei POPESCU <andreimpopescu@gmail.com> - 2021-05-17 14:10 +0200
                  Re: OT: minimum bs for dd? <tomas@tuxteam.de> - 2021-05-17 10:10 +0200
                    Re: OT: minimum bs for dd? Andrei POPESCU <andreimpopescu@gmail.com> - 2021-05-17 14:30 +0200
                      Re: OT: minimum bs for dd? <tomas@tuxteam.de> - 2021-05-17 15:00 +0200
                    info pages WHERE? -- was [Re: OT: minimum bs for dd?] Richard Owlett <rowlett@cloud85.net> - 2021-05-17 15:00 +0200
                      Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] <tomas@tuxteam.de> - 2021-05-17 15:10 +0200
                        Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] Richard Owlett <rowlett@cloud85.net> - 2021-05-17 15:20 +0200
                          Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] IL Ka <kazakevichilya@gmail.com> - 2021-05-17 15:30 +0200
                            Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] Richard Owlett <rowlett@cloud85.net> - 2021-05-17 16:00 +0200
                        Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] Greg Wooledge <greg@wooledge.org> - 2021-05-17 15:40 +0200
                          Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] "Thomas Schmitt" <scdbackup@gmx.net> - 2021-05-17 16:10 +0200
                            Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] Greg Wooledge <greg@wooledge.org> - 2021-05-17 17:10 +0200
                              Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] "Thomas Schmitt" <scdbackup@gmx.net> - 2021-05-17 18:00 +0200
                              Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] Brian <ad44@cityscape.co.uk> - 2021-05-17 20:30 +0200
                                Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] Greg Wooledge <greg@wooledge.org> - 2021-05-17 20:50 +0200
                                  Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] Brian <ad44@cityscape.co.uk> - 2021-05-17 21:00 +0200
                          Re: info pages WHERE? -- was [Re: OT: minimum bs for dd?] <tomas@tuxteam.de> - 2021-05-17 16:10 +0200
              Re: OT: minimum bs for dd? Brian <ad44@cityscape.co.uk> - 2021-05-17 16:20 +0200
            Re: OT: minimum bs for dd? <tomas@tuxteam.de> - 2021-05-17 08:40 +0200
              Re: OT: minimum bs for dd? Richard Hector <richard@walnut.gen.nz> - 2021-05-17 13:40 +0200
          Re: OT: minimum bs for dd? Michael Stone <mstone@debian.org> - 2021-05-16 20:50 +0200
            Re: OT: minimum bs for dd? "Andrew M.A. Cater" <amacater@einval.com> - 2021-05-16 23:40 +0200

Page 1 of 2  [1] 2  Next page →

#235221 — OT: minimum bs for dd?

As noted, is there a minimum bs size for dd?

Thank you.

-- 
I draw from the absurd three 
consequences, which are my revolt, 
my freedom, and my passion,

                       Camus

[toc] | [next] | [standalone]

#235222

[Multipart message — attachments visible in raw view] — view raw

On 2021-05-15 at 16:19, Bob Bernstein wrote:

> As noted, is there a minimum bs size for dd?

Yes: 1. dd will not accept a block size of 0, never mind a negative
block size. You can try it yourself:

$ dd if=/dev/zero of=/tmp/1zerobyte bs=1 count=1
1+0 records in
1+0 records out
1 byte copied, 0.000117071 s, 8.5 kB/s

That said, block sizes below a certain value will give you reduced
performance et cetera; what that value is depends on the physical
characteristics of the underlying hardware.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#235223

[Multipart message — attachments visible in raw view] — view raw

>
>
> As noted, is there a minimum bs size for dd?
>
It seems that you can set bs as small as 1.

512 is the default because of HDD block size which used to be 512 bytes for
more than 30 years (before advanced format was invented)



>

[toc] | [prev] | [next] | [standalone]

#235224

Thanks to all for your responses!

-- 
"No matter how big the problem is, you can always run away from it."
                        Dom Irrera

[toc] | [prev] | [next] | [standalone]

#235236

On Sat, May 15, 2021 at 11:32:31PM +0300, IL Ka wrote:
>
>    As noted, is there a minimum bs size for dd?
>
>It seems that you can set bs as small as 1.
>
>512 is the default because of HDD block size which used to be 512 bytes for
>more than 30 years (before advanced format was invented)

dd wasn't invented to do things with hard drives, it was for working
with tapes, file conversions, etc. Since record sizes on tapes and 
within files can vary widely, it's very flexible about what block sizes 
to use, and how to pad data to convert between block sizes. In most 
cases when people use dd these days it's not really the right tool for 
the job.

[toc] | [prev] | [next] | [standalone]

#235238

On 05/16/2021 12:42 PM, Michael Stone wrote:
> On Sat, May 15, 2021 at 11:32:31PM +0300, IL Ka wrote:
>>
>>    As noted, is there a minimum bs size for dd?
>>
>> It seems that you can set bs as small as 1.
>>
>> 512 is the default because of HDD block size which used to be 512 
>> bytes for
>> more than 30 years (before advanced format was invented)
> 
> dd wasn't invented to do things with hard drives, it was for working
> with tapes, file conversions, etc. Since record sizes on tapes and 
> within files can vary widely, it's very flexible about what block sizes 
> to use, and how to pad data to convert between block sizes. In most 
> cases when people use dd these days it's not really the right tool for 
> the job.

I'll bite ;}
When is it the right tool?
When is it not?

[toc] | [prev] | [next] | [standalone]

#235239

>> On Sun, May 16, 2021 at 01:31:49PM -0500, Richard Owlett wrote:
>> > I'll bite ;}
>> > When is it the right tool?
>> 
>> When you're using it to convert ebcdic to ascii, while swapping bytes and
>> reblocking an ancient file from a barely readable archival tape.
>> 
>> > When is it not?
>> 
>> When copying a file.
>> 
> When copying a file and writing it to another medium, perhaps eg when writing
> a DVD .iso file directly to a USB stick, it's ideal.

Not sure about ideal:

    cat <foo.iso >/dev/sdb

is one char longer than

    dd <foo.iso >/dev/sdb

but it's often faster (you can speed up `dd` by providing a larger
`bs=` argument, but then you've lost the length advantage ;-)


        Stefan

[toc] | [prev] | [next] | [standalone]

#235246

[Multipart message — attachments visible in raw view] — view raw

On 2021-05-16 at 18:33, Stefan Monnier wrote:

>> When copying a file and writing it to another medium, perhaps eg when writing
>> a DVD .iso file directly to a USB stick, it's ideal.
> 
> Not sure about ideal:
> 
>     cat <foo.iso >/dev/sdb
> 
> is one char longer than
> 
>     dd <foo.iso >/dev/sdb
> 
> but it's often faster (you can speed up `dd` by providing a larger
> `bs=` argument, but then you've lost the length advantage ;-)

Is there really no functional difference between the baseline trivial
functionalities of cat and dd?

From quite a long time ago, I thought there were contexts in which
writing a suitably-prepared data file directly to a block device using
cat would produce incorrect results, but using dd would work out fine. I
can't remember any of the potential specifics, though, so it's possible
I'm mixing one or both of them up with something else...

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#235251

Hi,

Andrew M.A. Cater wrote:
> > When copying a file and writing it to another medium, perhaps eg when
> > writing a DVD .iso file directly to a USB stick, it's ideal.

The Wanderer wrote:
> Is there really no functional difference between the baseline trivial
> functionalities of cat and dd?

cat and cp cannot do the following:

  # Remove possible GPT backup header from the end of the USB stick
  dd if=/dev/zero of=/dev/sdc bs=512 seek=3915775 count=1 status=none

  # Copy the ISO with its built-in partition table to the USB stick.
  # Show realistic progress messages by curbing the i/o buffering.
  ISO=debian-10.6.0-amd64-netinst.iso
  dd if="$ISO" of=/dev/'sdc' bs=1M status=progress oflag=dsync

With all copy methods one should do before unplugging the stick:

  sync


It should be noted that dd has its special pitfalls. E.g. if the input file
is not a data file or block device, then one should use
  iflag=fullblock
because with some versions of dd there is the risk to get partially read
blocks written as full output blocks.
(I cannot reproduce it right now by
   (echo hello ; sleep 1 ; echo hello) | dd of=dd_target_file obs=512
but i had undesirable results in the past.)


Whatever, the main risk with a short copy command performed by the superuser
is to pick the wrong device file name and thus to overwrite a disk with
valuable content.
I made a cautious (or paranoid) helper script for the task of copying ISOs
to USB sticks:
  https://wiki.debian.org/XorrisoDdTarget

It came too late for the freeze of Debian 11 but is planned to be
available as package in Debian 12. Up to then it can be downloaded from
upstream and verified by its corresponding .sig file:
  gpg --verify xorriso-dd-target.sig xorriso-dd-target


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#235260

[Multipart message — attachments visible in raw view] — view raw

On 2021-05-17 at 02:57, Thomas Schmitt wrote:

> Hi,
> 
> Andrew M.A. Cater wrote:
> 
>>> When copying a file and writing it to another medium, perhaps eg
>>> when writing a DVD .iso file directly to a USB stick, it's
>>> ideal.
> 
> The Wanderer wrote:
> 
>> Is there really no functional difference between the baseline
>> trivial functionalities of cat and dd?
> 
> cat and cp cannot do the following:
> 
>   # Remove possible GPT backup header from the end of the USB stick
>   dd if=/dev/zero of=/dev/sdc bs=512 seek=3915775 count=1 status=none

True, but that's not the baseline trivial functionality, which is what
Stefan had given in the example I was responding to; it includes a lot
of extra behavior-modifying flags.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#235261

On Sun, May 16, 2021 at 06:43:54PM -0400, The Wanderer wrote:
> >     cat <foo.iso >/dev/sdb
> > 
> >     dd <foo.iso >/dev/sdb

> Is there really no functional difference between the baseline trivial
> functionalities of cat and dd?

There are two differences:

1) dd is specified to use default input and output block sizes of 512 bytes.

2) dd is specified to write a summary to stderr upon completion.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html

[toc] | [prev] | [next] | [standalone]

#235248

[Multipart message — attachments visible in raw view] — view raw

On Du, 16 mai 21, 18:33:57, Stefan Monnier wrote:
> >> On Sun, May 16, 2021 at 01:31:49PM -0500, Richard Owlett wrote:
> >> > I'll bite ;}
> >> > When is it the right tool?
> >> 
> >> When you're using it to convert ebcdic to ascii, while swapping bytes and
> >> reblocking an ancient file from a barely readable archival tape.
> >> 
> >> > When is it not?
> >> 
> >> When copying a file.
> >> 
> > When copying a file and writing it to another medium, perhaps eg when writing
> > a DVD .iso file directly to a USB stick, it's ideal.
> 
> Not sure about ideal:
> 
>     cat <foo.iso >/dev/sdb
> 
> is one char longer than
> 
>     dd <foo.iso >/dev/sdb
> 
> but it's often faster (you can speed up `dd` by providing a larger
> `bs=` argument, but then you've lost the length advantage ;-)

I'll raise you 'cp':

    cp foo.iso /dev/sdb


which is both fast and short to type (apparently it's smart about using 
the correct block size).

Unfortunately it's missing dd's equivalent of status=progress.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

[toc] | [prev] | [next] | [standalone]

#235249

[Multipart message — attachments visible in raw view] — view raw

On Mon, May 17, 2021 at 08:59:43AM +0300, Andrei POPESCU wrote:

[...]

> I'll raise you 'cp':
> 
>     cp foo.iso /dev/sdb
> 
> 
> which is both fast and short to type (apparently it's smart about using 
> the correct block size).

I'll be sure to try that some day :)

> Unfortunately it's missing dd's equivalent of status=progress.

and oflags=sync.

Cheers
 - t

[toc] | [prev] | [next] | [standalone]

#235252

[Multipart message — attachments visible in raw view] — view raw

On Lu, 17 mai 21, 08:32:28, tomas@tuxteam.de wrote:
> On Mon, May 17, 2021 at 08:59:43AM +0300, Andrei POPESCU wrote:
> 
> [...]
> 
> > I'll raise you 'cp':
> > 
> >     cp foo.iso /dev/sdb
> > 
> > 
> > which is both fast and short to type (apparently it's smart about using 
> > the correct block size).
> 
> I'll be sure to try that some day :)
> 
> > Unfortunately it's missing dd's equivalent of status=progress.
> 
> and oflags=sync.

Hmm, would you (or anyone else) know what is the difference between 
oflags=sync and conv=fsync?

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

[toc] | [prev] | [next] | [standalone]

#235254

On Mon, 17 May 2021 at 17:29, Andrei POPESCU <andreimpopescu@gmail.com> wrote:

> Hmm, would you (or anyone else) know what is the difference between
> oflags=sync and conv=fsync?

https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html

https://unix.stackexchange.com/questions/508701/dd-command-oflag-direct-and-sync-flags

https://abbbi.github.io/dd/

[toc] | [prev] | [next] | [standalone]

#235265

[Multipart message — attachments visible in raw view] — view raw

On Lu, 17 mai 21, 17:50:24, David wrote:
> On Mon, 17 May 2021 at 17:29, Andrei POPESCU <andreimpopescu@gmail.com> wrote:
> 
> > Hmm, would you (or anyone else) know what is the difference between
> > oflags=sync and conv=fsync?
> 
> https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html
> 
> https://unix.stackexchange.com/questions/508701/dd-command-oflag-direct-and-sync-flags
> 
> https://abbbi.github.io/dd/
 
<blush>

Thank you, I should have guessed there is answer on StackExchange.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

[toc] | [prev] | [next] | [standalone]

#235255

[Multipart message — attachments visible in raw view] — view raw

On Mon, May 17, 2021 at 10:29:06AM +0300, Andrei POPESCU wrote:
> On Lu, 17 mai 21, 08:32:28, tomas@tuxteam.de wrote:
> > On Mon, May 17, 2021 at 08:59:43AM +0300, Andrei POPESCU wrote:
> > 
> > [...]
> > 
> > > I'll raise you 'cp':
> > > 
> > >     cp foo.iso /dev/sdb
> > > 
> > > 
> > > which is both fast and short to type (apparently it's smart about using 
> > > the correct block size).
> > 
> > I'll be sure to try that some day :)
> > 
> > > Unfortunately it's missing dd's equivalent of status=progress.
> > 
> > and oflags=sync.
> 
> Hmm, would you (or anyone else) know what is the difference between 
> oflags=sync and conv=fsync?

Let me put the docs next to each other (this is from the info page,
which sometimes is more complete than the man page):

* conv=fsync:

  ‘conv=CONVERSION[,CONVERSION]...’
     Convert the file as specified by the CONVERSION argument(s).  (No
     spaces around any comma(s).)

     [...]

     The following “conversions” are really file flags and don’t affect
     internal processing:

     [...]

     ‘fsync’
        Synchronize output data and metadata just before finishing.
        This forces a physical write of output data and metadata.

* oflag=sync:

  ‘oflag=FLAG[,FLAG]...’
     Access the output file using the flags specified by the FLAG
     argument(s).  (No spaces around any comma(s).)

     [...]

     ‘dsync’
        Use synchronized I/O for data.  For the output file, this
        forces a physical write of output data on each write.  For the
        input file, this flag can matter when reading from a remote
        file that has been written to synchronously by some other
        process.  Metadata (e.g., last-access and last-modified time)
        is not necessarily synchronized.

     ‘sync’
        Use synchronized I/O for both data and metadata.

I'd say "conv=fsync" does a sync at the end of the whole thing, while
"oflag=sync" open(2)s the file with O_SYNC (likewise "oflag=dsync"
translates to an open(2) with O_DSYNC), so the sync happens after each
block write. That's what I want, since it gives me much more predictability
as to when I can plop out my usb stick :)

Cheers
 - t

[toc] | [prev] | [next] | [standalone]

#235266

[Multipart message — attachments visible in raw view] — view raw

On Lu, 17 mai 21, 10:00:22, tomas@tuxteam.de wrote:
> On Mon, May 17, 2021 at 10:29:06AM +0300, Andrei POPESCU wrote:
> > 
> > Hmm, would you (or anyone else) know what is the difference between 
> > oflags=sync and conv=fsync?
> 
> Let me put the docs next to each other (this is from the info page,
> which sometimes is more complete than the man page):

[snip informative quote from info page]

Ugh, forgot about the info/man distinction fro GNU software.
 
> I'd say "conv=fsync" does a sync at the end of the whole thing, while
> "oflag=sync" open(2)s the file with O_SYNC (likewise "oflag=dsync"
> translates to an open(2) with O_DSYNC), so the sync happens after each
> block write. That's what I want, since it gives me much more predictability
> as to when I can plop out my usb stick :)

If my understanding from your quotes and David's links is correct 
oflag=sync may be slower in specific circumstances, but it depends on so 
many factors (hardware, caches, block size used, etc.) that it is hard 
to predict.

Quite likely it won't make a significant difference for "regular" use.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

[toc] | [prev] | [next] | [standalone]

#235271

[Multipart message — attachments visible in raw view] — view raw

On Mon, May 17, 2021 at 03:21:20PM +0300, Andrei POPESCU wrote:

[...]

> If my understanding from your quotes and David's links is correct 
> oflag=sync may be slower in specific circumstances, but it depends on so 
> many factors (hardware, caches, block size used, etc.) that it is hard 
> to predict.

The proof would be in the pudding, of course.

> Quite likely it won't make a significant difference for "regular" use.

For regular use, the oflag is (as I tried to explain) invaluable to me.

See, I bought myself a refurbished notebook, but I didn't skimp on
RAM: 16GB.

My main dd use is to write some image onto a flash removable (stick,
SD card). The write channel is slow, the available buffer big.

Once the write is done (if I do cp or forget the sync), not much
is actually written to the drive. Then I say "sync", and... the
sync goes shopping, without telling me when it plans to be back :-)

With "oflags=sync status=progress" (or by sending SIGHUP to the
dd process) I can get a rough idea when I'll be able to pull out
that stick.

Needless to say, I don't mind the copy being some, say, 10% slower
(totally making that up, but for a seq writing with some biggish
block size, I'd expect even that to be an outrageous exaggeration).

Cheers
 - t

[toc] | [prev] | [next] | [standalone]

#235273 — info pages WHERE? -- was [Re: OT: minimum bs for dd?]

On 05/17/2021 03:00 AM, tomas@tuxteam.de wrote:
> ... (this is from the info page, which sometimes is more complete than the man page):
> 

My web search turned up only compliments for info pages.
NOTHING on where to find :{
Help please.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web