Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #178510 > unrolled thread

Some help with dd backing up into an iso

Back to article view | Back to linux.debian.user

Contents

  Some help with dd backing up into an iso GiaThnYgeia <GiaThnYgeia@openmailbox.org> - 2017-03-07 01:20 +0100
    Re: Some help with dd backing up into an iso Michael Lange <klappnase@freenet.de> - 2017-03-07 02:10 +0100
    Re: Some help with dd backing up into an iso David Christensen <dpchrist@holgerdanske.com> - 2017-03-07 06:10 +0100
      Re: Some help with dd backing up into an iso David Christensen <dpchrist@holgerdanske.com> - 2017-03-07 07:00 +0100
        Re: Some help with dd backing up into an iso "Thomas Schmitt" <scdbackup@gmx.net> - 2017-03-07 09:00 +0100
          Re: Some help with dd backing up into an iso GiaThnYgeia <GiaThnYgeia@openmailbox.org> - 2017-03-07 18:30 +0100
            Re: Some help with dd backing up into an iso "Thomas Schmitt" <scdbackup@gmx.net> - 2017-03-07 20:20 +0100
              Re: Some help with dd backing up into an iso GiaThnYgeia <GiaThnYgeia@openmailbox.org> - 2017-03-08 05:00 +0100
                Re: Some help with dd backing up into an iso David Christensen <dpchrist@holgerdanske.com> - 2017-03-08 07:00 +0100
                Re: Some help with dd backing up into an iso "Thomas Schmitt" <scdbackup@gmx.net> - 2017-03-08 12:10 +0100
                  MBR partitioning, and content after partition table but before first  partition David Christensen <dpchrist@holgerdanske.com> - 2017-03-09 06:50 +0100
                    Re: MBR partitioning, and content after partition table but before  first partition Felix Miata <mrmazda@earthlink.net> - 2017-03-09 08:00 +0100
                      Re: MBR partitioning, and content after partition table but before  first partition David Christensen <dpchrist@holgerdanske.com> - 2017-03-10 06:10 +0100
                        Re: MBR partitioning, and content after partition table but before  first partition Jonathan Dowland <jmtd@debian.org> - 2017-03-10 10:00 +0100
                          Re: MBR partitioning, and content after partition table but before  first partition David Christensen <dpchrist@holgerdanske.com> - 2017-03-11 07:10 +0100
                            Re: MBR partitioning, and content after partition table but before  first partition Jonathan Dowland <jmtd@debian.org> - 2017-03-13 10:10 +0100
                              Re: MBR partitioning, and content after partition table but before  first partition David Christensen <dpchrist@holgerdanske.com> - 2017-03-14 04:40 +0100
                                Re: MBR partitioning, and content after partition table but before  first partition David <bouncingcats@gmail.com> - 2017-03-14 11:40 +0100
                                  Re: MBR partitioning, and content after partition table but before  first partition David Christensen <dpchrist@holgerdanske.com> - 2017-03-15 03:50 +0100
                                Re: MBR partitioning, and content after partition table but before  first partition The Wanderer <wanderer@fastmail.fm> - 2017-03-14 13:00 +0100
                                  Re: MBR partitioning, and content after partition table but before  first partition David Christensen <dpchrist@holgerdanske.com> - 2017-03-15 04:10 +0100
                                Re: MBR partitioning, and content after partition table but before  first partition Andy Smith <andy@strugglers.net> - 2017-03-15 04:40 +0100
                                Re: MBR partitioning, and content after partition table but before  first partition Jonathan Dowland <jmtd@debian.org> - 2017-03-15 13:00 +0100
                    Re: MBR partitioning, and content after partition table but before first partition "Thomas Schmitt" <scdbackup@gmx.net> - 2017-03-09 08:30 +0100
                    Re: MBR partitioning, and content after partition table but before  first partition Jonathan Dowland <jmtd@debian.org> - 2017-03-09 15:40 +0100
      Re: Some help with dd backing up into an iso Teemu Likonen <tlikonen@iki.fi> - 2017-03-07 09:30 +0100
        Re: Some help with dd backing up into an iso <tomas@tuxteam.de> - 2017-03-07 09:40 +0100
          Re: Some help with dd backing up into an iso Teemu Likonen <tlikonen@iki.fi> - 2017-03-07 10:00 +0100
            Re: Some help with dd backing up into an iso <tomas@tuxteam.de> - 2017-03-07 10:10 +0100
            Re: Some help with dd backing up into an iso David Wright <deblis@lionunicorn.co.uk> - 2017-03-07 16:20 +0100
        Re: Some help with dd backing up into an iso Jonathan Dowland <jmtd@debian.org> - 2017-03-07 11:10 +0100
        Re: Some help with dd backing up into an iso David Christensen <dpchrist@holgerdanske.com> - 2017-03-08 06:00 +0100
    Re: Some help with dd backing up into an iso Mark Fletcher <mark27q1@gmail.com> - 2017-03-07 12:10 +0100
    Re: Some help with dd backing up into an iso Eduardo M KALINOWSKI <eduardo@kalinowski.com.br> - 2017-03-07 14:50 +0100
    Re: Some help with dd backing up into an iso Elimar Riesebieter <riesebie@lxtec.de> - 2017-03-07 15:20 +0100
    Re: Some help with dd backing up into an iso David Wright <deblis@lionunicorn.co.uk> - 2017-03-07 16:20 +0100
      Re: Some help with dd backing up into an iso "Thomas Schmitt" <scdbackup@gmx.net> - 2017-03-07 18:10 +0100
        Re: Some help with dd backing up into an iso David Wright <deblis@lionunicorn.co.uk> - 2017-03-07 19:00 +0100
          Re: Some help with dd backing up into an iso "Thomas Schmitt" <scdbackup@gmx.net> - 2017-03-07 20:30 +0100
            Re: Some help with dd backing up into an iso David Wright <deblis@lionunicorn.co.uk> - 2017-03-07 21:30 +0100
              Re: Some help with dd backing up into an iso "Thomas Schmitt" <scdbackup@gmx.net> - 2017-03-07 22:10 +0100
                Re: Some help with dd backing up into an iso GiaThnYgeia <GiaThnYgeia@openmailbox.org> - 2017-03-08 18:30 +0100
                  Re: Some help with dd backing up into an iso David Wright <deblis@lionunicorn.co.uk> - 2017-03-08 19:20 +0100

Page 1 of 3  [1] 2 3  Next page →

#178510 — Some help with dd backing up into an iso

I am not very confident I am doing this right and it seems wrong,  I
can't locate any documentation that results into proper options.
I tried backing up an 8gb USB that has 2 partitions in it, one had 1.7gb
of data on it.
I used dd if=/dev/sdb of=usbfilename.iso
The resulting image was the full size of the disk.
To test the validity I restored reversing the order of the filenames
if/of  but that took for ever and it was a hog on resources.  After a
while I just gave up and killed the process.  I looked at the disk and
it seemed complete with all files in tact, so maybe I killed it
somewhere in the verification process.
So I used a program called etcher which I have used with 100% success in
the past and was surprisingly fast in burning images.
It took for ever as well, eventually it run a verification routine and
it was done.
Is there someway one can avoid creating such a large iso for no reason,
when the filesize is a fraction of the whole disk.  One way I thought of
was to shrink the partitions to just about 99% full, and leave the blank
part of the disk as not allocated.  Would that help?
Is there some fancy command line that does just that?

Thank you in advance




-- 
 "The most violent element in society is ignorance" rEG

[toc] | [next] | [standalone]

#178511

Hi,

On Tue, 07 Mar 2017 00:11:00 +0000
GiaThnYgeia <GiaThnYgeia@openmailbox.org> wrote:

> I am not very confident I am doing this right and it seems wrong,  I
> can't locate any documentation that results into proper options.
> I tried backing up an 8gb USB that has 2 partitions in it, one had 1.7gb
> of data on it.
> I used dd if=/dev/sdb of=usbfilename.iso
> The resulting image was the full size of the disk.
> To test the validity I restored reversing the order of the filenames
> if/of  but that took for ever and it was a hog on resources.  After a
> while I just gave up and killed the process.  I looked at the disk and
> it seemed complete with all files in tact, so maybe I killed it
> somewhere in the verification process.
> So I used a program called etcher which I have used with 100% success in
> the past and was surprisingly fast in burning images.
> It took for ever as well, eventually it run a verification routine and
> it was done.
> Is there someway one can avoid creating such a large iso for no reason,
> when the filesize is a fraction of the whole disk.  One way I thought of
> was to shrink the partitions to just about 99% full, and leave the blank
> part of the disk as not allocated.  Would that help?
> Is there some fancy command line that does just that?

you mean something like mkisofs? I cannot remember right now how the
cdrkit counterpart calls itself, but it probably does more or less the
same.

Regards

Michael


.-.. .. ...- .   .-.. --- -. --.   .- -. -..   .--. .-. --- ... .--. . .-.

	"What happened to the crewman?"
	"The M-5 computer needed a new power source, the crewman merely
         got in the way."
		-- Kirk and Dr. Richard Daystrom, "The Ultimate Computer",
		   stardate 4731.3.

[toc] | [prev] | [next] | [standalone]

#178518

On 03/06/2017 04:11 PM, GiaThnYgeia wrote:
> I am not very confident I am doing this right and it seems wrong,  I
> can't locate any documentation that results into proper options.
> I tried backing up an 8gb USB that has 2 partitions in it, one had 1.7gb
> of data on it.
> I used dd if=/dev/sdb of=usbfilename.iso
> The resulting image was the full size of the disk.
> To test the validity I restored reversing the order of the filenames
> if/of  but that took for ever and it was a hog on resources.  After a
> while I just gave up and killed the process.  I looked at the disk and
> it seemed complete with all files in tact, so maybe I killed it
> somewhere in the verification process.
> So I used a program called etcher which I have used with 100% success in
> the past and was surprisingly fast in burning images.
> It took for ever as well, eventually it run a verification routine and
> it was done.
> Is there someway one can avoid creating such a large iso for no reason,
> when the filesize is a fraction of the whole disk.  One way I thought of
> was to shrink the partitions to just about 99% full, and leave the blank
> part of the disk as not allocated.  Would that help?
> Is there some fancy command line that does just that?

Copying a raw device to a file, or vice-versa, I call "imaging".


ISO implies binary data structures ("format") on optical media so as to 
create a file system containing files and directories.


AFAIK USB flash drives use the same formats as hard disk drives and 
solid-state drives ("master boot record" (MBR) partition table, 
partition(s), and file system(s) within those partition(s)).


ISO and HDD/SSD formats are fundamentally different.  Taking an image of 
a USB flash drive and naming the output file with an *.iso extension 
will not translate the format to ISO.  Achieving that result requires 
tools other than 'dd'.


As for reducing the size of image files, the standard trick is to zero 
out the unused blocks first, and then compress the binary data stream 
while you take the image:

     # dd if=/dev/sda | gzip > myimage.img


For Linux and ext2, ext3, and ext4 file systems, the tool for zeroing 
unused blocks is zerofree(8):

https://manpages.debian.org/jessie/zerofree/zerofree.8.en.html


If you have an SSD, fstrim(8) will discard all unused blocks, regardless 
of file system.  They should then read as zeros:

https://manpages.debian.org/jessie/util-linux/fstrim.8.en.html


But, most USB flash drives come factory formatted with FAT32.  The 
'sdelete' Windows utility can be used to zero unused blocks for both 
NTFS and FAT file systems:

https://technet.microsoft.com/en-us/sysinternals/sdelete.aspx


David

[toc] | [prev] | [next] | [standalone]

#178519

On 03/06/2017 09:05 PM, David Christensen wrote:
> If you have an SSD, fstrim(8) will discard all unused blocks, regardless
> of file system.  They should then read as zeros:
>
> https://manpages.debian.org/jessie/util-linux/fstrim.8.en.html

I should qualify that:

     If you have an SSD, fstrim(8) will discard all unused blocks (for
     supported file systems and SSD's).


This computer has a SAMSUNG SSD UM410 Series 2.5" 16GB SSD with ext4 
boot and btrfs root file systems:

2017-03-06 21:48:24 root@jesse ~
# fstrim -v /boot
fstrim: /boot: the discard operation is not supported

2017-03-06 21:52:07 root@jesse ~
# fstrim -v /
fstrim: /: the discard operation is not supported


I need to test my other SSD's.


David

[toc] | [prev] | [next] | [standalone]

#178520

Hi,

GiaThnYgeia wrote:
> > I used dd if=/dev/sdb of=usbfilename.iso
> > The resulting image was the full size of the disk.

That's the job of dd: Copying block by block.

As David stated, the file usbfilename.iso will not be an ISO 9660 filesystem
but rather a disk image. 


David Christensen wrote:
> dd if=/dev/sda | gzip > myimage.img

The name suffix .img is more appropriate than .iso, indeed.
If your USB stick contains a lot of blocks with zeros, you may get even
better compression by using bzip2 instead of gzip:

  dd if=/dev/sdb | bzip2 >usbfilename.img

Zeroing the unused blocks before running dd will normally improve the
compression ratio and thus yield an even smaller .img file.

When you put it back on the USB stick, you need to uncompress:

  bunzip2 <usbfilename.img | dd of=/dev/sdb

This will recreate your two partitions and the filesystems which reside in
them.


> To test the validity I restored reversing the order of the filenames
> if/of  but that took for ever

By default dd copies chunks of 512 bytes.
Changing to 1 MiB chunks by dd option
  bs=1M
might speed up copying substantially.


> I looked at the disk and
> it seemed complete with all files in tact, so maybe I killed it
> somewhere in the verification process.

There is no verification process with plain dd.

Probably you successfully copied the first partition and the directory
tree of the second one. It has to be expected that not all files in the
second partition bear their original content, because it was not copied.


> Is there someway one can avoid creating such a large iso for no reason,
> when the filesize is a fraction of the whole disk.

That's called backup and archiving.
While the filesystem is mounted but fewly busy, you let a program read
the files and pack them up in that program's archive format.

The classic archiver is called "tar". Caution: wrong arguments or wrong
sequence of arguments can easily shoot your foot.
Assumed you have your filesystems mounted as 
  /mnt/usb_part1
  /mnt/usb_part2
you may pack them up gzip compressed by

  tar cvzf usb_part1.tar.gz /mnt/usb_part1
  tar cvzf usb_part2.tar.gz /mnt/usb_part2

The archive files usb_part1.tar.gz and usb_part2.tar.gz can be unpacked into
some directory by:

  cd /some/directory/for/part1
  tar xzf /where/it/is/usb_part1.tar.gz
  cd /some/directory/for/part2
  tar xzf /where/it/is/usb_part2.tar.gz


If the archive shall be an ISO 9660 filesystem, you may create it by

  xorriso -for_backup \
          -outdev usb_part1_and_2.iso \
          -map /mnt/usb_part1 /part1 \
          -map /mnt/usb_part2 /part2 \

The superuser will be able to mount the result for random access:

  mkdir /mnt/usb_par1_and_2
  mount -o loop /where/it/is/usb_part1_and2.iso /mnt/usb_par1_and_2

The normal user will then see and be able to read two file trees
  /mnt/usb_par1_and_2/part1
  /mnt/usb_par1_and_2/part2

To unmount the .iso, the superuser later does:

  umount /mnt/usb_par1_and_2


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#178548

I'd like to thank in advance ALL that responded, I think this is
valuable for an archive of a manual for the nearly illiterate.

Thomas Schmitt:
> GiaThnYgeia wrote:
>>> I used dd if=/dev/sdb of=usbfilename.iso
>>> The resulting image was the full size of the disk.
> 
> That's the job of dd: Copying block by block.
> 
> As David stated, the file usbfilename.iso will not be an ISO 9660 filesystem
> but rather a disk image. 

I have 3 encyclopedic questions based on this.  I noticed the difference
as I saved my first try.  With other "factory" iso images I can see most
of their contents with an archive program, this indicated a different
beast.  Although my restoration seemed to have worked fine but took for
ever.

1  So an img file does not matter what extension it has, it can be zzz
and it would be recognized as an image.  Correct?  The iso image is a
specific imaging system of cd/dvd format.

2.
  2.1  Block by block, and as it is mentioned in other responses about
zero-ing empty blocks, am I to understand correctly that erased data on
an empty block can be recovered because they are not zero.  Correct?

  2.2  If a block is zeroed it can't be unzeroed?  Is this what they
mean by deep cleaning/erasing?  That simple?  I remember in dos all it
took to make a block available was to take the first letter off of the
file name and it would vanish, unless someone put it back. UNDELETE!

3  If an encrypted partition is included in the image, empty or not, it
would be treated as a chunk of blocks that can't be altered without
being corrupted.  Correct?  So 4gb full of data in 1 partition with 4gb
of empty but encrypted partition is treated the same as 4gb full of
data,  Correct?

> David Christensen wrote:
>> dd if=/dev/sda | gzip > myimage.img
> 
> The name suffix .img is more appropriate than .iso, indeed.
> If your USB stick contains a lot of blocks with zeros, you may get even
> better compression by using bzip2 instead of gzip:
> 
>   dd if=/dev/sdb | bzip2 >usbfilename.img

So the of=usbfilename is replaced by the | bzip2?

> Zeroing the unused blocks before running dd will normally improve the
> compression ratio and thus yield an even smaller .img file.
> 
> When you put it back on the USB stick, you need to uncompress:
> 
>   bunzip2 <usbfilename.img | dd of=/dev/sdb

I will report back ... I'm willing to try this on my 1.8gb system on the
8gb stick.

> This will recreate your two partitions and the filesystems which reside in
> them.

You must be reading my mind, this is a bootable system, with a swap area
partition and the file system.  I have yet to see anything been written
in the swap area.  Is debian carrying this from an old functionality and
maybe due to systemd it is no longer being used?  Or it is getting used
only by certain applications.  I made it half the default size in case I
see it get used and be insufficient. I'm going to shrink it some more :)

> By default dd copies chunks of 512 bytes.
> Changing to 1 MiB chunks by dd option
>   bs=1M
> might speed up copying substantially.

1M used to be big!  I remember the excitement over the huge and mostly
unusable Double Density 1,4MB disks that replaced the paper 512KB double
sided monsters.  I wonder if kids would laugh at the sight of one.  I am
willing to bet that this dd goes back to backing up hard drives one
5.25" disk at the time.  And now we have 4mB empty files. I just created
out of curiosity an empty file and called it x.odt. It was 20bytes.  I
opened it with LibrOffice and saved it. It became an empty 10kB file.

>> I looked at the disk and
>> it seemed complete with all files in tact, so maybe I killed it
>> somewhere in the verification process.
> 
> There is no verification process with plain dd.

I assumed there might be because there was nothing messed up by my
premature interruption.  Etcher (https://etcher.io/) after finishing
went back and "verified" the whole thing, I guess the empty blocks as
well.  I suspect it is a fork of xorriso which I have downloaded but had
too many gaps in knowledge to understand how to use its options.
Etcher-electron is a tool for restoring images not for making them.
usbootin was too flaky and was dumped long ago.

> Probably you successfully copied the first partition and the directory
> tree of the second one. It has to be expected that not all files in the
> second partition bear their original content, because it was not copied.

True, if the data was less than 25% of the whole it was probably halfway
through copying the empty blocks!

>> Is there someway one can avoid creating such a large iso for no reason,
>> when the filesize is a fraction of the whole disk.
> 
> That's called backup and archiving.
> While the filesystem is mounted but fewly busy, you let a program read
> the files and pack them up in that program's archive format.

I may be unclear on this, my system was running on sda backing up a
different system on sdb, it wasn't archiving itself.  I thought that
this can't be done or shouldn't as the active system is not the same as
the shutdown-ed system.  I suspect this is not what you meant, you are
talking about used resources on sda trying to backup sdb?

> The classic archiver is called "tar". Caution: wrong arguments or wrong
> sequence of arguments can easily shoot your foot.

Never again!  I have the barrel pointing straight at the cpu!  Go ahead
punk!

> Assumed you have your filesystems mounted as 
>   /mnt/usb_part1
>   /mnt/usb_part2
> you may pack them up gzip compressed by
> 
>   tar cvzf usb_part1.tar.gz /mnt/usb_part1
>   tar cvzf usb_part2.tar.gz /mnt/usb_part2

But there is this gray area called usb_part(no number) which seems to
have some goodies that make parts 1 and 2 sing.  My formal training on
file systems goes back to stack of punched cards as input to huge
machines with huge tapes.  There were "computer offices" called RJE with
slaves working at the R(oman)emote Job Entry who handed out the output.
They were the monitors at the time and if they didn't cut off the one
output from the other some other student walked away with a box of your
work.  Past DOS6.22 I don't remember reading much about modern file
systems :) Let alone ext4. And it took a while to comprehend the boot
sector that I couldn't copy (I did eventually).

EFI and UEFI and BIOS-boot ... are still a hazy gib-rish symbolism that
I will not understand until I can mess with them enough and break them.
Theory was never good enough for me, I had to get the carburator off and
fix it to achieve perfect fuel mixture.  That resulted in wet plugs and
holes on pistons.  Now I know :)

> The archive files usb_part1.tar.gz and usb_part2.tar.gz can be unpacked into
> some directory by:
> 
>   cd /some/directory/for/part1
>   tar xzf /where/it/is/usb_part1.tar.gz
>   cd /some/directory/for/part2
>   tar xzf /where/it/is/usb_part2.tar.gz

That makes great sense, especially if you just want to back-up one
partition and not the whole thing.

> If the archive shall be an ISO 9660 filesystem, you may create it by
> 
>   xorriso -for_backup \
>           -outdev usb_part1_and_2.iso \
>           -map /mnt/usb_part1 /part1 \
>           -map /mnt/usb_part2 /part2 \

I gave up trying to understand all the tsorriso options, my brain turned
to jello, and just went on with dd (Default for Dummies) tool which took
my sleep away.  Just to come up with a valid command like what you say
one needs to spend a weekend studying Xausage or xorriso.

> The superuser will be able to mount the result for random access:
> 
>   mkdir /mnt/usb_par1_and_2
>   mount -o loop /where/it/is/usb_part1_and2.iso /mnt/usb_par1_and_2
> 
> The normal user will then see and be able to read two file trees
>   /mnt/usb_par1_and_2/part1
>   /mnt/usb_par1_and_2/part2

And that is how a network of dummy terminals that always boot up a fresh
installation of linux works.  No magic?

> To unmount the .iso, the superuser later does:
>   umount /mnt/usb_par1_and_2

Your time is up ... please insert 4 quarters for 15' more.

> Have a nice day :)
> Thomas

You have a teaching talent.... unlike the person who writes the
descriptions in the deb package archives who speaks in machine language

Most of them say: ZZZ is a tool for extracting YYY out of XXX in zyxxyz
system files so they can be translated to common xuzyzysyzyy archives
and be used by zyzyzyuxuxux systems!

WTFAYTA?

Imagine if English wasn't your native language!

Thanks again and to all that responded, there is material to read for a
weekend.

PS  There must be apple-msoft people out there laughing at all this for
just trying to make a backup of a usb stick.  Then they copy all this
free code, lock it in a fancy gui and sell it for billions.


-- 
 "The most violent element in society is ignorance" rEG

[toc] | [prev] | [next] | [standalone]

#178551

Hi,

GiaThnYgeia wrote:
> 1  So an img file does not matter what extension it has,

It's the data content which matters, not the name.
dd or cp don't care about name extensions.


> 2.1  Block by block, [...] erased data on
> an empty block can be recovered because they are not zero.  Correct?

If not the filesystem overwrote the content of deleted files
the problem will still be to find the content you are interested in.

The following run will show you cleartext snippets from the disk or image
in a text viewer which you can leave by pressing the "q" key:

  strings /dev/sdb | less

Now it depends on your knowledge about the desired content whether you
can find it.


> 2.2  If a block is zeroed it can't be unzeroed?

Not by normal means. You have to be aware that especially solid state
disks like USB keys have an own physical block management which might
delay the overwriting further.


> 3  If an encrypted partition is included in the image, empty or not, it
> would be treated as a chunk of blocks that can't be altered without
> being corrupted.  Correct?

It stays encrypted. In general it is dangerous to change blocks in a
disk image unless you understand fully what the data in the block mean.
The risk is high that you break the filesystem and will get errors
or wrong file data when you mount it.


> So 4gb full of data in 1 partition with 4gb
> of empty but encrypted partition is treated the same as 4gb full of
> data,  Correct?

Well, an encrypted partition should be much less compressable than an
empty unencrypted one. Encryption shall camouflage the true content.
So it can hardly represent all zeros as a similarly redundant byte set.


> > dd if=/dev/sdb | bzip2 >usbfilename.img
> So the of=usbfilename is replaced by the | bzip2?

Yes. If dd has no of= argument then it writes its data to standard
output, which is normally your terminal window.

But "|" establishes a pipe. It connects standard output of dd with
standard input of program bzip2. (That standard input would normally
be your keyboard and its Enter key.)

Since bzip2 gets no file name argument it reads from its standard input
and writes the compression result to its standard output.

But the ">" redirects the standard output of bzip2 to the data file
usbfilename.img.
So you do not get printed a lot of text salad on your terminal but
there rather emerges a data file with compressed content.

This connecting and redirecting of output is done by the shell, not by
the programs dd and bzip2.


> > When you put it back on the USB stick, you need to uncompress:
> I will report back ... I'm willing to try this on my 1.8gb system

Be careful not to spoil irrepairable data.


> I have yet to see anything been written
> in the swap area.

Maybe your computer has lots of RAM or the swap is not in use ?
(What does shell command "free" report ? Is ther a line starting with
 "Swap:" and giving three numbers ?)


> maybe due to systemd it is no longer being used?

Systemd is a convenient suspect for everything. But i doubt that it
can make swap space obsolete when the RAM does not suffice.


> 1M used to be big!

Yeah ... Love, 36 bit, and punched cards ...


> I am willing to bet that this dd goes back to backing up hard drives

Its origin is in IBM's Job Control Language. From there it came to early
Unix when there were still unused combinations of two letters. "cp", "ls",
"dd", "cc", "ld" ... "cat" is of course an example of wastefulness.


> Etcher (https://etcher.io/) after finishing
> went back and "verified" the whole thing,

A good idea to do so.


> I suspect it is a fork of xorriso 

No. xorriso packs up files as inhabitants of an ISO 9660 filesystem.
There are no forks known.


> had too many gaps in knowledge to understand how to use its options.

If you tell me the path to the mount point of the data partition i will
modify my previous example to that address.


> > While the filesystem is mounted but fewly busy, you let a program read
> > the files and pack them up in that program's archive format.

> I may be unclear on this, my system was running on sda backing up a
> different system on sdb, it wasn't archiving itself.

That's very wise. Making a dd copy of an active system disk will at best
cause the symptoms of a heavy system power failure when you restore the
system to a disk and then try to start it up.

Backup programs usually run on mounted filesystems and thus are prone
to recording inconsistent file states if files change while the backup
is run. This danger can be avoided entirely if the filesystem is mounted
read-only.


> > Assumed you have your filesystems mounted as 
> >   /mnt/usb_part1
> >   /mnt/usb_part2

> But there is this gray area called usb_part(no number) which seems to
> have some goodies that make parts 1 and 2 sing.

Now i know that one of them is swap and thus not mountable.
But the other one is supposed to be mounted in your overall filesystem.
Either by an automounter (e.g. systemd-udev) or by a mount command issued
by the superuser (i.e. you with your sudo hat on).

You will have to find out that path to the root of your partition's
filesystem. Then you can backup it.


> I gave up trying to understand all the tsorriso options,

Nobody is supposed to. Just collect in a shell script what you need and
what you learn from the internet, man page examples, or me.


> And that is how a network of dummy terminals that always boot up a fresh
> installation of linux works.  No magic?

My xorriso example (now to be done with only one partition) is for making
a backup of the partition filesystem. You can access the files by mounting
the ISO filesystem or by using one of the archivers which can read ISO 9660.
(Mounting is a feature of the operating system.)

The magic gate for booting is in the boot sectors which let the firmware
start the very magic boot loader which then starts the incredibly magic
operating system.
I only know about boot sectors. The rest is the job of e.g. debian-cd which
prepares the files of boot loader and operating system, and the job of
the boot loader which has to deal with the hardware on which it wakes up.
 

> You have a teaching talent...

Despite i am the one who wrote the man page of xorriso ?


> Most of them say: ZZZ is a tool for [...]

It is quite difficult to describe to users something that you know on
source code level.


> Imagine if English wasn't your native language!

This might be another part of the problem. I'm german.


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#178561

Thomas Schmitt:
> Hi,
> 
> GiaThnYgeia wrote:
>> 2.1  Block by block, [...] erased data on
>> an empty block can be recovered because they are not zero.  Correct?
> 
> If not the filesystem overwrote the content of deleted files
> the problem will still be to find the content you are interested in.

Good to know, but no I am not looking for deleted data.

>> 2.2  If a block is zeroed it can't be unzeroed?
> 
> Not by normal means. You have to be aware that especially solid state
> disks like USB keys have an own physical block management which might
> delay the overwriting further.

So each manufacturer may have a different internal system but the output
is standardized.  So we don't really know what goes on in there, right?
I had gone down the isle of trying to read chips and modifying them (1
to break printer out of ink pseudo blocks - 2 to look into internal
ignition/injection maps for vehicle modification) but I gave up early
for not been able to locate proper instruments.  One of the reasons that
I got back on computing and away from windowz.

>> So 4gb full of data in 1 partition with 4gb
>> of empty but encrypted partition is treated the same as 4gb full of
>> data,  Correct?
> 
> Well, an encrypted partition should be much less compressable than an
> empty unencrypted one. Encryption shall camouflage the true content.
> So it can hardly represent all zeros as a similarly redundant byte set.

Aaahh.. so part of the trick is filling in fake data so you can't tell
the real ones, I get it.  I think!

>>> dd if=/dev/sdb | bzip2 >usbfilename.img
>> So the of=usbfilename is replaced by the | bzip2?
> 
> Yes. If dd has no of= argument then it writes its data to standard
> output, which is normally your terminal window.

The matrix :)  The real thing.

> But "|" establishes a pipe. It connects standard output of dd with
> standard input of program bzip2. (That standard input would normally
> be your keyboard and its Enter key.)
> Since bzip2 gets no file name argument it reads from its standard input
> and writes the compression result to its standard output.
> But the ">" redirects the standard output of bzip2 to the data file
> usbfilename.img.
> So you do not get printed a lot of text salad on your terminal but
> there rather emerges a data file with compressed content.

OK, perfect sense.

> This connecting and redirecting of output is done by the shell, not by
> the programs dd and bzip2.

So it is like combining various processes and their inputs and outputs
into one, which is the final product the user needs.

>>> When you put it back on the USB stick, you need to uncompress:
>> I will report back ... I'm willing to try this on my 1.8gb system
> 
> Be careful not to spoil irrepairable data.

Well at this point it is all experimentation so I know how to do it
right when I need to.

>> I have yet to see anything been written
>> in the swap area.
> 
> Maybe your computer has lots of RAM or the swap is not in use ?
> (What does shell command "free" report ? Is ther a line starting with
>  "Swap:" and giving three numbers ?)

I didn't know of this command, all I could see was the partition always
being empty. I suppose things get written and deleted (swapped) only
when RAmemory runs out.  And I thought only ms-win did such silly things.

>> maybe due to systemd it is no longer being used?
> 
> Systemd is a convenient suspect for everything. But i doubt that it
> can make swap space obsolete when the RAM does not suffice.

$free
              total        used        free      shared  buff/cache
available
Mem:        3884232     1596976      302428      135400     1984828
1864344
Swap:             0           0           0

Tried it on a stressed out 32bit VBox system and  it was not 0 0 0  but
I had only given it 1gB of RAM.

>> 1M used to be big!
> 
> Yeah ... Love, 36 bit, and punched cards ...

Before that in a machine shop CNC machine I wrote code into a paper tape
with 5 columns of holes. Like a 70s Telex machine
In the mid-90s I took a turn to work OUT with hands and tools and no
digits.  Now I am back at easy comfortable air/conditioned life, red-hat
is still around, and multi-processing things and satas and all kinds of
crazy stuff I need to catch up with.

>> I am willing to bet that this dd goes back to backing up hard drives
> 
> Its origin is in IBM's Job Control Language. From there it came to early
> Unix when there were still unused combinations of two letters. "cp", "ls",
> "dd", "cc", "ld" ... "cat" is of course an example of wastefulness.

I never touched any VMs ... the machine owners were DEC customers, and
we did alot of work on VT100s with real hard keys.

>> Etcher (https://etcher.io/) after finishing
>> went back and "verified" the whole thing,
> 
> A good idea to do so.

So my headache is to create a proper image and in an efficient way,
restoring it seems easy.

>> I suspect it is a fork of xorriso 
> 
> No. xorriso packs up files as inhabitants of an ISO 9660 filesystem.
> There are no forks known.

Yes, I thought the problem with dd was that I didn't know how to tell it
to use this ISO9660 that I thought was the problem of not being able to
read the img as an archive. In other isos I've seen you can see the
filesystem partition.  So I started studying xorriso but I got lost with
all the different options that had little meaning to me.

>> had too many gaps in knowledge to understand how to use its options.
> 
> If you tell me the path to the mount point of the data partition i will
> modify my previous example to that address.

depending on the position or how many sticks I have on it is usually
dev/sdb or dev/sdc As filemanagers like pcmanfm do not reveal the right
partition name only the label, I use gparted to make sure I have the
right one.

>>> While the filesystem is mounted but fewly busy, you let a program read
>>> the files and pack them up in that program's archive format.
> 
>> I may be unclear on this, my system was running on sda backing up a
>> different system on sdb, it wasn't archiving itself.
> 
> That's very wise. Making a dd copy of an active system disk will at best
> cause the symptoms of a heavy system power failure when you restore the
> system to a disk and then try to start it up.

At best I suppose it would be like trying to start something that is
already running.

> Backup programs usually run on mounted filesystems and thus are prone
> to recording inconsistent file states if files change while the backup
> is run. This danger can be avoided entirely if the filesystem is mounted
> read-only.

Since you have been willing to teach at the elementary level I'll shoot
for more.  Once you install a system like debian, does the information
in the hidden part of the disk ever change, of can I just copy the file
system partition as a backup and replace it if it breaks?

>>> Assumed you have your filesystems mounted as 
>>>   /mnt/usb_part1
>>>   /mnt/usb_part2
> 
>> But there is this gray area called usb_part(no number) which seems to
>> have some goodies that make parts 1 and 2 sing.
> 
> Now i know that one of them is swap and thus not mountable.
> But the other one is supposed to be mounted in your overall filesystem.
> Either by an automounter (e.g. systemd-udev) or by a mount command issued
> by the superuser (i.e. you with your sudo hat on).

This systemd-udev is not within the filesystem partition, it is outside
somewhere in the beggining of the disk, which is what "mounts" the file
system and gets it running. Correct? What is the boot flag for and is it
necessary on linux?  Or is the boot flag only needed when there is
nothing else telling bios to start that specific partition and run?

> You will have to find out that path to the root of your partition's
> filesystem. Then you can backup it.

OK, let's say swap is sdb1 and the filesystem is sdb2 (for this example
there is no other partition). There are about 2mb in the beggining of
the sdb then 2 partitions.  This is an actual working installation not a
live system.  With the exception of having to reinstall sound drivers I
got it working on 2 different but similar systems.  I was surprised it
did, but it does.  This may help me from having to maintain two separate
but parallel systems and just have one portable one and just use the hd
for just data files of work that I do.

>> I gave up trying to understand all the tsorriso options,
> 
> Nobody is supposed to. Just collect in a shell script what you need and
> what you learn from the internet, man page examples, or me.

:)  I should have looked at the name of the maintainer up before I open
my big mouth :)  Nice to meet you mr Libburnia :)

>> And that is how a network of dummy terminals that always boot up a fresh
>> installation of linux works.  No magic?
> 
> My xorriso example (now to be done with only one partition) is for making
> a backup of the partition filesystem. You can access the files by mounting
> the ISO filesystem or by using one of the archivers which can read ISO 9660.
> (Mounting is a feature of the operating system.)

Many of the examples I found were specific to partitions and not a whole
bootable disk.  Unless one studies filesystems it is hard to understand
why this 9660 is important.  What I understood is its limitation to long
and complex filenames.  So if I was to back up something with huge
filenames I suspect it may run into problems of altering them and not
being able to restore them correctly.  Have I made stew out of what I read?

> The magic gate for booting is in the boot sectors which let the firmware
> start the very magic boot loader which then starts the incredibly magic
> operating system.
> I only know about boot sectors. The rest is the job of e.g. debian-cd which
> prepares the files of boot loader and operating system, and the job of
> the boot loader which has to deal with the hardware on which it wakes up.

I always wondered why a machine with no-hd and a cd drive, would have to
go on and run on an error before you can press the button, open the
door,insert the cd, and have to restart it.  I think at some point that
changed into an error (no disk found, insert new disk and press enter)
or it was variation of bios setups.
My specific question with this is how often in a debian system do things
in the boot sector change, if ever.  I can't believe that if you have a
Debian2 and you change the source to jessie the bootloader stays the
same of it is the same with entering a fresh install of 8.7.1

>> You have a teaching talent...
> 
> Despite i am the one who wrote the man page of xorriso ?

Well, ... I guess it is different when writing a manual that you think
only developers and sys-admins will ever look at, who understand all
those funky terms, than when you have to explain it to a guitarist that
only cares about backing up his composition, recordings, midi files into
a stick and transport it to the studio.  If the studio only uses some
commercial expensive apple stuff it helps  having a file-system with you
to edit the files while on the run.

>> Most of them say: ZZZ is a tool for [...]
> 
> It is quite difficult to describe to users something that you know on
> source code level.

This is a trend with any tech/science field today.  Only experts of the
same specialty can understand each other.  If you go back and read the
notes of technicians (applied physicists before engineering was a
science) of around WW1 era, things that may even seem advanced and
unpatented yet are written in language that can be understood by a wide
background.  Today if a biologist is working on human skin tissue and
another is working on eye-cells they understand nothing of each other's
work.  So science is totally useless to everyone but the employers of
scientists.  What you are admitting here is being a bi-product of that
same system.  And we need to change this, otherwise your great xorriso
code may only be useful to some SONY/SAMSUNG who may manufacture a tiny
little backup machine with closed up code to sell to people not like me,
who actually do have the money to buy it.

>> Imagine if English wasn't your native language!
> 
> This might be another part of the problem. I'm german.

I meant for those who read the manuals generally for linux and apart
from the terminology their skills in English are limited as well.

> Have a nice day :)
> Thomas

You too, maybe I can take a more careful look at the site and help with
the descriptions and instructions from a different perspective. I am
trying to make sense out of all this stuff so I can create instructions
for another language.


-- 
 "The most violent element in society is ignorance" rEG

[toc] | [prev] | [next] | [standalone]

#178565

On 03/07/2017 07:53 PM, GiaThnYgeia wrote:
> Once you install a system like debian, does the information
> in the hidden part of the disk ever change, of can I just copy the file
> system partition as a backup and replace it if it breaks?

AFAIK when using MBR partitioning, the partition table (blocks 0-62) 
does not change if the partitions are not changed (start sector, size, 
type, flags, etc.).


> OK, let's say swap is sdb1 and the filesystem is sdb2 (for this example
> there is no other partition). There are about 2mb in the beggining of
> the sdb then 2 partitions.  This is an actual working installation not a
> live system.  With the exception of having to reinstall sound drivers I
> got it working on 2 different but similar systems.  I was surprised it
> did, but it does.  This may help me from having to maintain two separate
> but parallel systems and just have one portable one and just use the hd
> for just data files of work that I do.

I discovered that I can copy a Wheezy system drive image from an HDD/SSD 
to a USB flash drive, and the USB flash drive will then boot and run in 
several of my computers.  I have not tried it with Jesse yet, but expect 
it will work.


> Many of the examples I found were specific to partitions and not a whole
> bootable disk.  Unless one studies filesystems it is hard to understand
> why this 9660 is important.  What I understood is its limitation to long
> and complex filenames.  So if I was to back up something with huge
> filenames I suspect it may run into problems of altering them and not
> being able to restore them correctly.  Have I made stew out of what I read?

If you want to burn files and directories to optical disc (I call this 
"archiving"), I suggest you start by using a simple GUI tool such as Xfburn.


> My specific question with this is how often in a debian system do things
> in the boot sector change, if ever.  I can't believe that if you have a
> Debian2 and you change the source to jessie the bootloader stays the
> same of it is the same with entering a fresh install of 8.7.1

The way to find out if the partition table changes between Debian 
releases would be to install a release of Debian, take an image of the 
first 2048 sectors of the system drive, hex dump it, install another 
release of Debian, take an image of the first 2048 sectors, hex dump 
that, and then run 'diff' on the two hex dumps.


> ... guitarist that
> only cares about backing up his composition, recordings, midi files into
> a stick and transport it to the studio.  If the studio only uses some
> commercial expensive apple stuff it helps  having a file-system with you
> to edit the files while on the run.

Having working copies of files on multiple disks is a double-edged 
sword.  It works best if you only read the files.  If and when you 
modify a file, now you have to figure out how to synchronize it to all 
the other disks.  The risk is that you'll edit a given file in two or 
more places, and then have to solve the three way merge problem:

https://en.wikipedia.org/wiki/Merge_%28version_control%29#Three-way_merge


A version control system is very useful for keeping things straight, and 
warning you when you have a problem.  I use CVS.  The repository is 
stored on my file server.  I can check out, update, and commit files on 
whatever computer and disk I want.  The trick is to always update before 
you work on files, and always commit when you are done.


David

[toc] | [prev] | [next] | [standalone]

#178573

Hi,

GiaThnYgeia wrote:
> So each manufacturer may have a different internal system but the output
> is standardized.  So we don't really know what goes on in there, right?

Yes. We programmers enjoy the simplified model of an array of consecutive
logical blocks. The physical blocks are a matter of disk manufacturer and
data rescue companies.


> > Encryption shall camouflage the true content.

> so part of the trick is filling in fake data so you can't tell
> the real ones,

It is not necessarily adding of fake data but rather exchanging
understandable or predictable bit patterns by other bit patterns which
we would consider random noise.


> > [shell piping and redirection]

> So it is like combining various processes and their inputs and outputs
> into one, which is the final product the user needs.

Yep. This is what makes the colorful GUIs look like fancy toys
- if not like handcuffs.


> $free
> ...
> Mem:        3884232     1596976      302428   ... 
> Swap:             0           0           0

There is no swap space enabled. And your RAM is not really large.
You should look for advise about system startup and swap configuration.


> In other isos I've seen you can see the filesystem partition.
> So I started studying xorriso

Partitions inside an ISO 9660 filesystems are normally preparations for
booting from USB stick. Quite a variform topic:
  https://dev.lovelyhq.com/libburnia/libisofs/blob/master/doc/boot_sectors.txt


> > If you tell me the path to the mount point of the data partition i will
> > modify my previous example to that address.

> depending on the position or how many sticks I have on it is usually
> dev/sdb or dev/sdc

That's the device path, not the mount point.
The mount point is a directory in the overall filesystem tree. With removable
media it is nowadays created underneath /media or /mnt. This directory then
represents the root directory of the mounted additional filesystem.


> Once you install a system like debian, does the information
> in the hidden part of the disk ever change,

Well, on the level of logical blocks there is no hidden part.

There may be space which is not claimed by partitions.
The unclaimed space before the first partition is usually inhabited by
a partition table and possibly by boot loader software.
Changes of partitioning or boot loader will happen there.
A GUID partition table (GPT) has a backup copy at the very end of the
storage medium, which is not part of any partition and changes in sync
with the main GPT table at the start of the medium.


> This systemd-udev is not within the filesystem partition, it is outside
> somewhere in the beggining of the disk [...] ?

No. systemd is (if ever) part of the operating system that gets started
by the boot loader. This start procedure starts a program (the kernel of
the operating system), gives it a minimal filesystem (initial RAM disk),
and tells it the disk partition which shall serve as root filesystem.

>From then on the started operating system is in charge.
At some stage of starting up it will start the systemd program which then
may be in charge of mounting other filesystems into the root filesystem
tree.

But here my knowledge gets thin. Maybe Michael Biebl can explain better
when and how systemd starts up and how it is related to mounting and
activating swap.


> What is the boot flag for and is it necessary on linux?

The Boot Flag is a bit in the partition table of an MBR.
It is simply a indicator for some boot loaders, from which partition
they shall load the program code for the next step of booting.

This does not apply to the two main boot loaders used with Linux on
BIOS firmware: SYSLINUX/ISOLINUX and GRUB. Those loaders know by other
means from where to load the info and programs for their next steps
of booting.

Some firmwares, especially brain damaged EFI implementations, insist
in seeing the boot flag at some MBR partition or else they will not
consider the whole device as bootable.
On the other hand the EFI System Partiton must not be marked by the
boot flag. So EFI from MBR partition table needs a second partition
to expose the boot flag to the poorly implemented EFIs.


> let's say swap is sdb1 and the filesystem is sdb2

For a backup or archiver program you need the mount point path, not the
partition path.
On conservatively mounting systems you may look up the mount point
address by searching for the partition name:

  mount | grep sdb2

will reply something like

  /dev/sdb2 on /tmp type ext4 (rw,relatime,data=ordered)

Here /tmp is the mount point.

Less conservative systems mount by partition UUID rather than by
partition path. So no "sdb2" will be to find in the output of command
"mount". Nevertheless, "mount" without "| grep sdb2" will tell you
all mount points. Maybe you can identify your partition in that list. 


> [portable data disk]  may help me from having to maintain two separate
> but parallel systems and just have one portable one and just use the hd
> for just data files of work that I do.

As long as the filesystem type (ext4, btrfs, ...) is supported by the
operating systems it should be no problem to read and (if applicable)
write data to it.
If there is system software in the filesystem it may be that this runs
as program only on the system by which it was installed in the filesystem. 
But those program files can still be copied, overwritten, or renamed.


> Unless one studies filesystems it is hard to understand
> why this 9660 is important.

It is mountable by about all known operating systems. Further it is
to be mounted read-only. So it is a good candidate for archiving and backup.

There is a special boot sector format prescribed for PC-BIOS and EFI
when the boot medium is a CD, DVD, or Blu-ray disk. This format is named
El Torito (after the restaurants) and specified as add-on to ISO 9660
filesystems.
So about every bootable optical disc is equipped with an ISO 9660
filesystem (possibly augmented by UDF filesystem, or Joliet directory
trees).


> What I understood is its limitation to long and complex filenames.
> So if I was to back up something with huge filenames I suspect it
> may run into problems

The filenames of ISO 9660 are indeed not better than with MS-DOS. 
But there is the Rock Ridge extension (named after the movie Blazing
Saddles) which records file names of up to 255 characters length and
distinguishes between upper case and lower case chracters. I.e. it
fulfills the demands of X/Open in respect to filesystem naming schemes
on a Unix-like system.

Reading Rock Ridge is a speciality of Unix-like operating systems.
On a non-Unix operating system, the file names may indeed appear
crippled to short length and all upper or all lower case.
Microsoft has its own add-on named "Joliet". Names up to 64 characters.
(xorriso command -joliet "on" activates its production.)

xoRRiso has Rock Ridge in its name and in its genes. So don't worry
about backup fidelity as long as you want to restore it on a Linux
system. (xorriso command -for_backup will also store ACLs and Extended
attributes, but only xorriso can recognize them in the ISO and restore
them by its extraction commands.)


> how often in a debian system do things
> in the boot sector change, if ever.

Normally the unclaimed regions of a storage medium are left unchanged.
But installing or updating a boot loader will change the start region.
Editing the partition table will too cause a change there.


> If the studio only uses some
> commercial expensive apple stuff it helps  having a file-system with you
> to edit the files while on the run.

One may change the effective content of an ISO 9660 filesystem but
this is not as easy as with a read-write filesystem.
If you want read-write and maximum compatibility to operating systems,
the old FAT filesystem is the best choice. (Not for archiving, to be clear.)
That's the filesystem type which you normally find on newly purchased
USB sticks.


David Christensen wrote:
> AFAIK when using MBR partitioning, the partition table (blocks 0-62)

The MBR partition table resides in the first 512-bytes block.
It may be extended by a chain of partitions starting at the Extended
Partition of the MBR partition table.
The EBRs of the logical partitions are stored at the start of those
partitions. I.e. scattered over the storage medium.

A GPT partition table begins at the second 512-block and can stretch
over quite a large range of blocks, depending on the number of
allocated partition slots. 4 partition slots occupy one 512-block.


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#178593 — MBR partitioning, and content after partition table but before first partition

On 03/08/2017 03:02 AM, Thomas Schmitt wrote:
> David Christensen wrote:
>> AFAIK when using MBR partitioning, the partition table (blocks 0-62)
>
> The MBR partition table resides in the first 512-bytes block.
> It may be extended by a chain of partitions starting at the Extended
> Partition of the MBR partition table.
> The EBRs of the logical partitions are stored at the start of those
> partitions. I.e. scattered over the storage medium.

Okay.


Examining a Windows XP disk, the first partition (C:\) starts at block 
63 (track 1):

$ cat windows-xp-parted-u-s-p.out
Model: ATA WDC WD800JB-00FM (scsi)
Disk /dev/sdb: 156301488s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start  End         Size        Type     File system  Flags
  1      63s    156296384s  156296322s  primary  ntfs         boot


$ cat windows-xp-parted-u-chs-p.out
Model: ATA WDC WD800JB-00FM (scsi)
Disk /dev/sdb: 9729,80,62
Sector size (logical/physical): 512B/512B
BIOS cylinder,head,sector geometry: 9729,255,63.  Each cylinder is 8225kB.
Partition Table: msdos

Number  Start  End          Type     File system  Flags
  1      0,1,0  9728,254,62  primary  ntfs         boot


Blocks 1-62 should probably be zeros, but it appears Norton Ghost 2003 
marked this drive:

$ dd if=windows-xp-blocks-0-62.img 2>/dev/null | hexdump -C | tail -n 12
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa 
|..............U.|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
*
00007c00  12 91 f2 60 90 a0 2f 19  01 00 00 00 85 00 67 68 
|...`../.......gh|
00007c10  46 44 23 cc 22 52 1d ac  22 52 32 4e 6f 72 74 6f 
|FD#."R.."R2Norto|
00007c20  6e 20 47 68 6f 73 74 20  32 30 30 33 00 00 00 00  |n Ghost 
2003....|
00007c30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
*
00007c90  00 00 00 00 00 00 00 04  00 94 2d d0 80 00 00 00 
|..........-.....|
00007ca0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
*
00007e00


Examining a Jesse system drive, the first partition starts at block 2048 
(1 MB = 2**20 bytes):

2017-03-08 21:30:04 root@jesse ~
# parted /dev/sda u s p
Model: ATA SAMSUNG SSD UM41 (scsi)
Disk /dev/sda: 31277232s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start     End        Size       Type     File system  Flags
  1      2048s     976895s    974848s    primary  ext4         boot
  2      976896s   1953791s   976896s    primary
  3      1953792s  28125183s  26171392s  primary


There is content in the first 101 blocks (1 MBR plus 100 other):

2017-03-08 21:30:06 root@jesse ~
# dd if=/dev/sda count=2048 2>/dev/null | hexdump -C | tail
0000c990  b0 b3 2c fa a4 38 f1 f8  77 f0 2d dd c2 4a a4 a9 
|..,..8..w.-..J..|
0000c9a0  cd 43 15 5a b7 43 1e 6b  8b 01 ad 6f a0 b9 63 fd 
|.C.Z.C.k...o..c.|
0000c9b0  b8 59 4d 69 f4 33 d4 f4  0a c1 d0 61 df 84 a9 74 
|.YMi.3.....a...t|
0000c9c0  64 44 ae 2d de bc c5 de  73 90 d8 94 ef b2 5a 7c 
|dD.-....s.....Z||
0000c9d0  fe 4c ee d8 a0 d7 ce ae  28 72 cc 5b 7e 05 23 ee 
|.L......(r.[~.#.|
0000c9e0  39 d0 44 7a 1c 0a 10 ab  e4 f6 3b 6f 43 d1 e6 92 
|9.Dz......;oC...|
0000c9f0  11 0b 3f ec c4 cf b1 ab  81 b9 5b 77 d8 be 90 c6 
|..?.......[w....|
0000ca00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
*
00100000


What is in blocks 1-101?


David

[toc] | [prev] | [next] | [standalone]

#178595 — Re: MBR partitioning, and content after partition table but before first partition

David Christensen composed on 2017-03-08 21:46 (UTC-0800):
...
> Examining a Jesse system drive, the first partition starts at block 2048
> (1 MB = 2**20 bytes):

> 2017-03-08 21:30:04 root@jesse ~
> # parted /dev/sda u s p
> Model: ATA SAMSUNG SSD UM41 (scsi)
> Disk /dev/sda: 31277232s
> Sector size (logical/physical): 512B/512B
> Partition Table: msdos
> Disk Flags:

> Number  Start     End        Size       Type     File system  Flags
>   1      2048s     976895s    974848s    primary  ext4         boot
>   2      976896s   1953791s   976896s    primary
>   3      1953792s  28125183s  26171392s  primary

> There is content in the first 101 blocks (1 MBR plus 100 other):

> 2017-03-08 21:30:06 root@jesse ~
> # dd if=/dev/sda count=2048 2>/dev/null | hexdump -C | tail
...

> What is in blocks 1-101?

Was that disk ever used for anything besides Jessie, not new or wiped first?

Run strings on it or view in a sector editor and you'll probably see grub 
somewhere, if it's a typical Linux installation that puts Grub in the MBR 
instead of on a primary partition with its boot flag set, along with generic 
code in the MBR.

Occasionally some of those sectors are used for RAID.
-- 
"The wise are known for their understanding, and pleasant
words are persuasive." Proverbs 16:21 (New Living Translation)

  Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata  ***  http://fm.no-ip.com/

[toc] | [prev] | [next] | [standalone]

#178631 — Re: MBR partitioning, and content after partition table but before first partition

On 03/08/2017 10:56 PM, Felix Miata wrote:
> Was that disk ever used for anything besides Jessie, not new or
> wiped first?

The disk was wiped before installing Jesse.


> Run strings on it or view in a sector editor and you'll probably see
> grub somewhere, if it's a typical Linux installation that puts Grub
> in the MBR instead of on a primary partition with its boot flag set,
> along with generic code in the MBR.

2017-03-09 20:22:00 root@jesse ~
# dd if=/dev/sda skip=1 count=100 2>/dev/null | file -
/dev/stdin: data

2017-03-09 20:22:38 root@jesse ~
# dd if=/dev/sda skip=1 count=100 2>/dev/null | strings | egrep -i
'[a-z]{5,}'
loading
  Error
RBRPQR
&EhALJ
RcVqt
YpmBs
0GXRSh
rswcGnj
#NXxVg
ihfer
zxgKNZT
LsELn
UenvkO
kxgqI&
MstUFX1
g~vqKsu8


> Occasionally some of those sectors are used for RAID.

I don't use RAID with that system disk.


On 03/08/2017 11:21 PM, Thomas Schmitt wrote:
> Could be GRUB program code. Do you see cleartext messages in the
> first 100 blocks ?

See above.


On 03/09/2017 06:30 AM, Jonathan Dowland wrote:
> I believe it's part of grub. My limited understanding of how it works
> is it's split up into separate stages designed to fit within the
> "holes" in a typical MBR layout, each stage having enough code to
> initialise some stuff and then jump to the next stage.


I use LUKS swap (random key) and root (passphrase).  I think it's the 
piece of the boot chain that gives me the LUKS prompt for root (before 
the GRUB menu).


My Wheezy system drive also uses LUKS:

2017-03-09 20:32:04 root@cd2533 ~
# dd if=/dev/sdb count=2048 2>/dev/null | hexdump -C | tail
0000cd90  00 0f 7d 74 a2 57 6f 16  d8 61 5a 19 6d c0 63 f3 
|..}t.Wo..aZ.m.c.|
0000cda0  ed 86 77 f3 9d 2a 26 87  40 61 e3 7a 05 a7 fc b0 
|..w..*&.@a.z....|
0000cdb0  8b 5d e1 b3 99 a4 ad 17  8e f6 dd 5f 1e 1a 06 5d 
|.]........._...]|
0000cdc0  8f f3 e9 b6 ec f7 63 18  72 47 d4 aa 3b 5a da 56 
|......c.rG..;Z.V|
0000cdd0  ac 2d 95 3a c4 12 53 45  6a 5a 92 21 79 6c d6 ca 
|.-.:..SEjZ.!yl..|
0000cde0  93 ec ae fe 96 c3 df cb  61 ca 88 d8 a0 14 7d 53 
|........a.....}S|
0000cdf0  31 c4 53 ba 2e aa ee 35  57 46 fd 4b 81 9c 8d 9d 
|1.S....5WF.K....|
0000ce00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
*
00100000

2017-03-09 20:34:43 root@cd2533 ~
# dd if=/dev/sdb skip=1 count=2048 2>/dev/null | file -
/dev/stdin: data

2017-03-09 20:36:49 root@cd2533 ~
# dd if=/dev/sdb skip=1 count=2048 2>/dev/null | strings | egrep -i 
'[a-z]{5,}'
loading
  Error
RBRPQR
wPdwm
piyEst
WmZFH7w
ukdmtI
kAXwU
NQapDd
wkNdR
MKEZH'
GZRCv
vNigR
VpbVk
mXDbl
l8WMhMJ
5ygCtG
PjzpIj$3
flnYG


So, the moral of the story appears to be:

     When taking an image of a Debian system drive, be sure to copy the
     blocks between the partition table and the first partition, as there
     may be boot loader code there.


I can find plenty of "introductory" / "overview" documents, but does 
anybody know if/ where the technical details are documented?


2017-03-09 20:46:57 dpchrist@jesse ~
$ apt-get source grub-pc
...

2017-03-09 20:58:13 dpchrist@jesse ~
$ cd grub2-2.02~beta2/docs/

2017-03-09 20:58:17 dpchrist@jesse ~/grub2-2.02~beta2/docs
$ ls -w 72
Makefile.am	       grub-dev.texi  stamp-1
Makefile.in	       grub.cfg       stamp-vti
autoiso.cfg	       grub.info      texinfo.tex
fdl.texi	       grub.texi      version-dev.texi
font_char_metrics.png  man	      version.texi
font_char_metrics.txt  mdate-sh
grub-dev.info	       osdetect.cfg


This document is dated and doesn't cover LUKS, but does indicate that 
blocks 1-62 were used for "GRUB stage 1.5":

http://www.pixelbeat.org/docs/disk/


David

[toc] | [prev] | [next] | [standalone]

#178635 — Re: MBR partitioning, and content after partition table but before first partition

[Multipart message — attachments visible in raw view] — view raw

On Thu, Mar 09, 2017 at 09:04:56PM -0800, David Christensen wrote:
> I use LUKS swap (random key) and root (passphrase).  I think it's the piece
> of the boot chain that gives me the LUKS prompt for root (before the GRUB
> menu).

You get that prompt *before* GRUB?

I use LUKS everywhere and only get it after, because the prompt is (normally)
issued from my initrd. (although I historically put /boot outside of the
encryption. Maybe this is some newer scheme I am not familiar with).

> So, the moral of the story appears to be:
> 
>     When taking an image of a Debian system drive, be sure to copy the
>     blocks between the partition table and the first partition, as there
>     may be boot loader code there.

I'd always put a step 0) in there: is imaging what you want to do? Consider
a file-level backup with rsync (etc etc, as discussed elsewhere in this 
thread)
 
> This document is dated and doesn't cover LUKS, but does indicate that blocks
> 1-62 were used for "GRUB stage 1.5":
> 
> http://www.pixelbeat.org/docs/disk/

Yes, that was what I thought.

-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.

[toc] | [prev] | [next] | [standalone]

#178667 — Re: MBR partitioning, and content after partition table but before first partition

On 03/10/2017 12:49 AM, Jonathan Dowland wrote:
> On Thu, Mar 09, 2017 at 09:04:56PM -0800, David Christensen wrote:
>> I use LUKS swap (random key) and root (passphrase).  I think it's the piece
>> of the boot chain that gives me the LUKS prompt for root (before the GRUB
>> menu).
>
> You get that prompt *before* GRUB?
>
> I use LUKS everywhere and only get it after, because the prompt is (normally)
> issued from my initrd.

You're right, I mis-remembered -- I get the Grub menu and then the LUKS 
passphrase prompt for root.


> (although I historically put /boot outside of the
> encryption. Maybe this is some newer scheme I am not familiar with).

My boot partition is also unencrypted.


>> So, the moral of the story appears to be:
>>
>>     When taking an image of a Debian system drive, be sure to copy the
>>     blocks between the partition table and the first partition, as there
>>     may be boot loader code there.
>
> I'd always put a step 0) in there: is imaging what you want to do? Consider
> a file-level backup with rsync (etc etc, as discussed elsewhere in this
> thread)

I do imaging for system disks.  I do backups and archives for data.


David

[toc] | [prev] | [next] | [standalone]

#178747 — Re: MBR partitioning, and content after partition table but before first partition

[Multipart message — attachments visible in raw view] — view raw

On Fri, Mar 10, 2017 at 10:00:45PM -0800, David Christensen wrote:
> >I'd always put a step 0) in there: is imaging what you want to do? Consider
> >a file-level backup with rsync (etc etc, as discussed elsewhere in this
> >thread)
> 
> I do imaging for system disks.  I do backups and archives for data.

So having evangelised file-level copies a few times in this thread, I found
myself wondering if I would have been better off with imaging this very
weekend. Copying a 2.1T filesystem from an internal SATA2 disk to an external
one (my regular backup drive to my once-a-month, lives off-site one) via USB3
took nearly 48 hours via "rsync -a", and the destination ended up bigger,
possibly because one or more of the backups on the source had been using some
kind of hardlink de-dupe (I've ranted about hardlink trees being a problem in
various backup topics on -user, too...) and I didn't think to supply -S to
rsync.

The real test will be how long an incremental catch-up will take in the future.

-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.

[toc] | [prev] | [next] | [standalone]

#178808 — Re: MBR partitioning, and content after partition table but before first partition

On 03/13/2017 02:01 AM, Jonathan Dowland wrote:
> On Fri, Mar 10, 2017 at 10:00:45PM -0800, David Christensen wrote:
>>> I'd always put a step 0) in there: is imaging what you want to do? Consider
>>> a file-level backup with rsync (etc etc, as discussed elsewhere in this
>>> thread)
>>
>> I do imaging for system disks.  I do backups and archives for data.
>
> So having evangelised file-level copies a few times in this thread, I found
> myself wondering if I would have been better off with imaging this very
> weekend. Copying a 2.1T filesystem from an internal SATA2 disk to an external
> one (my regular backup drive to my once-a-month, lives off-site one) via USB3
> took nearly 48 hours via "rsync -a",

2.1 TB / 48 hr / 3600 s/hr = 12.2 MB/s

I was also disappointed by the transfer rate of external USB drives on 
Debian.  Firewire is better.  eSATA is best.


I now use 3 TB Seagate ST3000DM001 desktop drives in StarTech 
DRW115SATBK mobile docks connected to motherboard and/or HBA SATA ports. 
  With LUKS and a Pentium D 945 (no AES-NI), I see 40 MB/s.  With LUKS 
and a Core i7-2600S (AES-NI), I see 220 MB/s.


> and the destination ended up bigger,
> possibly because one or more of the backups on the source had been using some
> kind of hardlink de-dupe (I've ranted about hardlink trees being a problem in
> various backup topics on -user, too...) and I didn't think to supply -S to
> rsync.

-S is for sparse files.


Doing a quick test, it appears that rsync copies hard linked files as if 
each were a different file:

2017-03-13 20:33:46 dpchrist@jesse ~/sandbox/rsync
$ cat hard-link
#!/bin/sh
# Test 'rsync -a' and hard links
# $Id: hard-link,v 1.2 2017/03/14 03:33:15 dpchrist Exp $
# by David Paul Christensen dpchrist@holgerdanske.com
# Public Domain
rm -rf hard-link-1
rm -rf hard-link-2
mkdir hard-link-1
mkdir hard-link-2
echo "hello, world!" > hard-link-1/hello.txt
ln hard-link-1/hello.txt hard-link-1/link-1.txt
ln hard-link-1/hello.txt hard-link-1/link-2.txt
ln hard-link-1/hello.txt hard-link-1/link-3.txt
ln hard-link-1/hello.txt hard-link-1/link-4.txt
ls -li hard-link-1/*
du -b hard-link-1/*
rsync -a hard-link-1/ hard-link-2
ls -li hard-link-2/*
du -b hard-link-2/*

2017-03-13 20:34:18 dpchrist@jesse ~/sandbox/rsync
$ sh hard-link
271759 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 13 20:34 hard-link-1/hello.txt
271759 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 13 20:34 hard-link-1/link-1.txt
271759 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 13 20:34 hard-link-1/link-2.txt
271759 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 13 20:34 hard-link-1/link-3.txt
271759 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 13 20:34 hard-link-1/link-4.txt
14	hard-link-1/hello.txt
271760 -rw-r--r-- 1 dpchrist dpchrist 14 Mar 13 20:34 hard-link-2/hello.txt
271761 -rw-r--r-- 1 dpchrist dpchrist 14 Mar 13 20:34 hard-link-2/link-1.txt
271762 -rw-r--r-- 1 dpchrist dpchrist 14 Mar 13 20:34 hard-link-2/link-2.txt
271763 -rw-r--r-- 1 dpchrist dpchrist 14 Mar 13 20:34 hard-link-2/link-3.txt
271764 -rw-r--r-- 1 dpchrist dpchrist 14 Mar 13 20:34 hard-link-2/link-4.txt
14	hard-link-2/hello.txt
14	hard-link-2/link-1.txt
14	hard-link-2/link-2.txt
14	hard-link-2/link-3.txt
14	hard-link-2/link-4.txt


Is anyone aware of a utility that can walk a file system and replace 
identical files with hard links?


> The real test will be how long an incremental catch-up will take in the future.

For new large files, the size of the files divided by 12.2 MB/s.  For 
everything else, longer.


David

[toc] | [prev] | [next] | [standalone]

#178817 — Re: MBR partitioning, and content after partition table but before first partition

On 14 March 2017 at 14:36, David Christensen <dpchrist@holgerdanske.com> wrote:
>
> Doing a quick test, it appears that rsync copies hard linked files as if
> each were a different file:
>
> rsync -a hard-link-1/ hard-link-2

Here, 'man rsync' says:
"Note that -a does not preserve hardlinks, because finding
multiply-linked files is expensive.  You must separately specify -H."

[toc] | [prev] | [next] | [standalone]

#178850 — Re: MBR partitioning, and content after partition table but before first partition

On 03/14/2017 03:34 AM, David wrote:
> On 14 March 2017 at 14:36, David Christensen <dpchrist@holgerdanske.com> wrote:
>>
>> Doing a quick test, it appears that rsync copies hard linked files as if
>> each were a different file:
>>
>> rsync -a hard-link-1/ hard-link-2
>
> Here, 'man rsync' says:
> "Note that -a does not preserve hardlinks, because finding
> multiply-linked files is expensive.  You must separately specify -H."

Thanks for the tip.  :-)


So, rsync can copy hard links:

2017-03-14 19:33:10 dpchrist@jesse ~/sandbox/rsync
$ cat hard-link
#!/bin/sh
# Test 'rsync -a' and hard links
# $Id: hard-link,v 1.3 2017/03/15 02:32:08 dpchrist Exp $
# by David Paul Christensen dpchrist@holgerdanske.com
# Public Domain
mkdir hard-link-1
mkdir hard-link-2
echo "hello, world!" > hard-link-1/hello.txt
ln hard-link-1/hello.txt hard-link-1/link-1.txt
ln hard-link-1/hello.txt hard-link-1/link-2.txt
ln hard-link-1/hello.txt hard-link-1/link-3.txt
ln hard-link-1/hello.txt hard-link-1/link-4.txt
ls -li hard-link-1/*
du -b hard-link-1/*
rsync -a -H hard-link-1/ hard-link-2
ls -li hard-link-2/*
du -b hard-link-2/*
rm -rf hard-link-1
rm -rf hard-link-2


2017-03-14 19:33:13 dpchrist@jesse ~/sandbox/rsync
$ sh hard-link
272911 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 14 19:33 hard-link-1/hello.txt
272911 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 14 19:33 hard-link-1/link-1.txt
272911 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 14 19:33 hard-link-1/link-2.txt
272911 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 14 19:33 hard-link-1/link-3.txt
272911 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 14 19:33 hard-link-1/link-4.txt
14	hard-link-1/hello.txt
272912 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 14 19:33 hard-link-2/hello.txt
272912 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 14 19:33 hard-link-2/link-1.txt
272912 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 14 19:33 hard-link-2/link-2.txt
272912 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 14 19:33 hard-link-2/link-3.txt
272912 -rw-r--r-- 5 dpchrist dpchrist 14 Mar 14 19:33 hard-link-2/link-4.txt
14	hard-link-2/hello.txt


On 03/14/2017 04:52 AM, The Wanderer wrote:
 > (You may want to check whether the resulting files are hardlinked back
 > to the ones in the original tree; I haven't tested, and from reading the
 > man page it doesn't seem entirely clear.)


The test above indicates that hard links created by rsync with the -H 
option point to files within the destination tree.


David

[toc] | [prev] | [next] | [standalone]

#178822 — Re: MBR partitioning, and content after partition table but before first partition

[Multipart message — attachments visible in raw view] — view raw

On 2017-03-13 at 23:36, David Christensen wrote:

> On 03/13/2017 02:01 AM, Jonathan Dowland wrote:
> 
>> On Fri, Mar 10, 2017 at 10:00:45PM -0800, David Christensen wrote:

>> and the destination ended up bigger, possibly because one or more
>> of the backups on the source had been using some kind of hardlink
>> de-dupe (I've ranted about hardlink trees being a problem in 
>> various backup topics on -user, too...) and I didn't think to
>> supply -S to rsync.
> 
> -S is for sparse files.
> 
> 
> Doing a quick test, it appears that rsync copies hard linked files as
> if each were a different file:

<snip>

As already mentioned, you need the '-H' option to rsync for that. My
standard rsync invocation is with '-avPH', just in case the tree being
copied has any hardlinks.

(You may want to check whether the resulting files are hardlinked back
to the ones in the original tree; I haven't tested, and from reading the
man page it doesn't seem entirely clear.)

> Is anyone aware of a utility that can walk a file system and replace
>  identical files with hard links?

Try rdfind. It's in Debian; I don't use it myself, largely because the
(accepted upstream years ago) feature request for a "-minsize" option
(to replace or extend the "-ignoreempty" option) which I need for my use
case has not apparently been implemented yet - but finding duplicate
files is exactly what it's meant for, and one of its available "actions"
is to replace the duplicate files with hardlinks.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.debian.user

csiph-web