Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > linux.debian.user > #265915 > unrolled thread

512e vs 4K sector confusion

Started byAndy Smith <andy@strugglers.net>
First post2024-01-14 09:10 +0100
Last post2024-01-20 22:30 +0100
Articles 16 — 9 participants

Back to article view | Back to linux.debian.user


Contents

  512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-14 09:10 +0100
    Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-14 09:20 +0100
      Re: 512e vs 4K sector confusion "Alexander V. Makartsev" <avbetev@gmail.com> - 2024-01-14 12:30 +0100
      Re: 512e vs 4K sector confusion Arno Lehmann <al@its-lehmann.de> - 2024-01-14 12:40 +0100
    Re: 512e vs 4K sector confusion "Thomas Schmitt" <scdbackup@gmx.net> - 2024-01-14 14:30 +0100
      Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-14 17:20 +0100
        Re: 512e vs 4K sector confusion "Thomas Schmitt" <scdbackup@gmx.net> - 2024-01-14 18:00 +0100
    Re: 512e vs 4K sector confusion Timothy M Butterworth <timothy.m.butterworth@gmail.com> - 2024-01-14 19:20 +0100
    Re: 512e vs 4K sector confusion Stefan Monnier <monnier@iro.umontreal.ca> - 2024-01-15 05:40 +0100
      Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-15 06:40 +0100
        Re: 512e vs 4K sector confusion Nicolas George <george@nsup.org> - 2024-01-15 22:30 +0100
        Re: 512e vs 4K sector confusion Nicholas Geovanis <nickgeovanis@gmail.com> - 2024-01-15 22:30 +0100
          Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-16 01:40 +0100
          Re: 512e vs 4K sector confusion Teemu Likonen <tlikonen@iki.fi> - 2024-01-16 17:40 +0100
        Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-17 08:30 +0100
    Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-20 22:30 +0100

#265915 — 512e vs 4K sector confusion

FromAndy Smith <andy@strugglers.net>
Date2024-01-14 09:10 +0100
Subject512e vs 4K sector confusion
Message-ID<HW3J7-2ZIb-5@gated-at.bofh.it>
Hi,

I've got a disk image that sits on top of an LVM logical volume
that is on top of an mdadm RAID-1 that is on top of a pair of:

Device Model:     Samsung SSD 870 EVO 4TB
Sector Size:      512 bytes logical/physical

so let';s say that is at /dev/foo/disk_image (where /dev/foo is the
name of the LVM VG and disk_image is the LV)

So,

# fdisk -ul /dev/foo/disk_image
Disk /dev/foo/disk_image: 400 GiB, 429496729600 bytes, 838860800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x14409245

Device                            Boot Start       End   Sectors  Size Id Type
/dev/foo_disk_image                    2048 838858751 838856704  400G 83 Linux

So it's a 400G disk image with an MBR and a single partition, right?

Now, I dd that disk image across the network to another machine
which has a similar setup, except here the "foo" volume group is on
a pair of

Device Model:     HGST HUS726T6TALN6L4
Sector Size:      4096 bytes logical/physical

Now, after the disk_image has arrived, it looks very odd. fdisk
thinks it is 8 times bigger than it really is, and thinks it has 4K
sectors. I can't use "kpartx" to get at the partition inside it, and
fsck.ext4 doesn't like its first partition at all.

Is there any way to make this work?

If necessary and if there is a way, I *can* nuke off the target
machine's "foo" volume group and recreate the RAID array if I have
to make it 512e format. But obviously I'd like some way to move this
disk image and have it still work without having to meddle inside it
much — it is a VM disk.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [next] | [standalone]


#265917

FromAndy Smith <andy@strugglers.net>
Date2024-01-14 09:20 +0100
Message-ID<HW3SN-2ZLr-11@gated-at.bofh.it>
In reply to#265915
On Sun, Jan 14, 2024 at 08:01:52AM +0000, Andy Smith wrote:
> If necessary and if there is a way, I *can* nuke off the target
> machine's "foo" volume group and recreate the RAID array if I have
> to make it 512e format. But obviously I'd like some way to move this
> disk image and have it still work without having to meddle inside it
> much — it is a VM disk.

I think I may be able to use hdparm to reformat these Ultrastar DCs
from 4kn to 512e…

hdparm --set-sector-size 512 --please-destroy-my-drive /dev/sdX

https://wiki.archlinux.org/title/Advanced_Format#Advanced_Format_hard_disk_drives

It is okay to destroy the drive contents as there's nothing working
on there yet given this initial failure.

I think that is the only way this is going to work without mounting
filesystems both sides and using an fs-aware tool like tar|ssh or
rsync… which I don't want to do.

Thoughts?

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]


#265923

From"Alexander V. Makartsev" <avbetev@gmail.com>
Date2024-01-14 12:30 +0100
Message-ID<HW6QF-31w6-7@gated-at.bofh.it>
In reply to#265917

[Multipart message — attachments visible in raw view] — view raw

On 14.01.2024 13:15, Andy Smith wrote:
> On Sun, Jan 14, 2024 at 08:01:52AM +0000, Andy Smith wrote:
>> If necessary and if there is a way, I *can* nuke off the target
>> machine's "foo" volume group and recreate the RAID array if I have
>> to make it 512e format. But obviously I'd like some way to move this
>> disk image and have it still work without having to meddle inside it
>> much — it is a VM disk.
> I think I may be able to use hdparm to reformat these Ultrastar DCs
> from 4kn to 512e…
>
> hdparm --set-sector-size 512 --please-destroy-my-drive /dev/sdX
>
> https://wiki.archlinux.org/title/Advanced_Format#Advanced_Format_hard_disk_drives
>
> It is okay to destroy the drive contents as there's nothing working
> on there yet given this initial failure.
>
> I think that is the only way this is going to work without mounting
> filesystems both sides and using an fs-aware tool like tar|ssh or
> rsync… which I don't want to do.
>
> Thoughts?
Weird. I can imagine some wild theories. :)
I think it has something to do with a fact that "disk_image" is a LVM 
volume and you dd copy "/dev/mapper/VG--name-LV--name" not "/dev/sdc2" .
Could it be that "dd" sees source and destination as block devices and 
copies data by sector? I.e. takes 512b sector and place it to the 
beginning of 4096b sector and padding the rest 3584b of the sector with 
zeroes.
This calls for some tests. I'd create 1G image test file with dos 
partition table and a few primary partitions:

    $ dd if=/dev/zero of=./1G-disk-image.bin bs=1M count=1000
    1000+0 records in
    1000+0 records out
    1048576000 bytes (1,0 GB, 1000 MiB) copied, 8,07768 s, 130 MB/s

    ...
    $ /sbin/fdisk -l ./1G-disk-image.bin
    Disk ./1G-disk-image.bin: 1000 MiB, 1048576000 bytes, 2048000 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0x143acb59

    Device               Boot   Start     End Sectors  Size Id Type
    ./1G-disk-image.bin1         2048  411647  409600  200M 83 Linux
    ./1G-disk-image.bin2       411648 1435647 1024000  500M 83 Linux
    ./1G-disk-image.bin3      1435648 2047999  612352  299M 83 Linux


And dd copied it the same way to disk with 4k sectors.
Just to see if it will be inconsistent too, without LVM mapper involved.

I'd also try to clone disk to image file first, with say "partclone" 
utility, "scp" the image file and restore it at remote destination with 
the same utility after recreating dumped partition table layout using 
"sfdisk".
I did this procedure multiple times successfully, but without 4k sector 
disks and LVM involvement.


-- 
With kindest regards, Alexander.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀

[toc] | [prev] | [next] | [standalone]


#265924

FromArno Lehmann <al@its-lehmann.de>
Date2024-01-14 12:40 +0100
Message-ID<HW70m-31z9-3@gated-at.bofh.it>
In reply to#265917
Hi andy,

Am 14.01.2024 um 09:15 schrieb Andy Smith:
> On Sun, Jan 14, 2024 at 08:01:52AM +0000, Andy Smith wrote:
>> If necessary and if there is a way, I *can* nuke off the target
>> machine's "foo" volume group and recreate the RAID array if I have
>> to make it 512e format. But obviously I'd like some way to move this
>> disk image and have it still work without having to meddle inside it
>> much — it is a VM disk.
> 
> I think I may be able to use hdparm to reformat these Ultrastar DCs
> from 4kn to 512e…
...
> Thoughts?


Reshaping the target system's storage may be the quickest and most 
reliable solution.

If you expect there may be future cases with similar problems -- the 
other way around would be much worse, i.e. you have a source storage 
with 4k sectors, and target that does not support it -- these posts

https://unix.stackexchange.com/questions/748208/lvm-and-device-mapper-logical-volume-device-sector-size

https://unix.stackexchange.com/questions/676534/creating-an-ssd-partition-with-a-different-block-size

might be good starting points to work out a real solution.

As far as I know, however, it is not possible to change the block sizes 
using any of the software layers involved -- they all inherit from their 
base layer, which means from the actual devices, eventually.

Fortunately, I never had to solve such issues myself, but that's 
probably because I prefer working at the file system level. Block level 
just has too many pitfalls for my simple tastes ;-)

I'm getting curious to see what happens if I have to mix 512- and 
4k-sector disks in one RAID, though.

Cheers,

Arno


-- 
Arno Lehmann

IT-Service Lehmann
Sandstr. 6, 49080 Osnabrück

[toc] | [prev] | [next] | [standalone]


#265927

From"Thomas Schmitt" <scdbackup@gmx.net>
Date2024-01-14 14:30 +0100
Message-ID<HW8IN-32CB-7@gated-at.bofh.it>
In reply to#265915
Hi,

Andy Smith wrote:
> I've got a disk image that sits on top of an LVM logical volume
> that is on top of an mdadm RAID-1 that is on top of a pair of:
> Device Model:     Samsung SSD 870 EVO 4TB
> Sector Size:      512 bytes logical/physical
> so let';s say that is at /dev/foo/disk_image (where /dev/foo is the
> name of the LVM VG and disk_image is the LV)
> So
> # fdisk -ul /dev/foo/disk_image
> Disk /dev/foo/disk_image: 400 GiB, 429496729600 bytes, 838860800 sectors
> Units: sectors of 1 * 512 = 512 bytes

fdisk thinks that this device has a logical block size of 512.

This information is not stored in MBR partition table (or in GPT) but
rather it is a property of the storage device.


> Disklabel type: dos
> So it's a 400G disk image with an MBR and a single partition, right?

Yes.

(But nitpicking:
It has an MBR partition table with entries other than the protective
entry which would indicate the presence of GPT. I.e. GPT contains an
MBR too, but its partition table entries are stored outside of the MBR.)


> Now, I dd that disk image across the network to another machine
> which has a similar setup, except here the "foo" volume group is on
> a pair of
> Device Model:     HGST HUS726T6TALN6L4
> Sector Size:      4096 bytes logical/physical

So the logical volume was probably created with the logical block size
of the involved physical disk devices.


> fdisk
> thinks it is 8 times bigger than it really is, and thinks it has 4K
> sectors.

fdisk asks the device (unless it is a regular file, i guess) for its
logical size. Since the partition table gives no hint about its own
idea of the bytes/block ratio, you end up with vastely inflated byte
numbers.

I wonder, though, why fdisk would misrepresent the total disk size,
which i would expect to come from the storage device.
Maybe fdisk is more confused than i expect.

Can you show the whole output of fdisk from the new logical device ?


Whatever:
Not only the partition table might get misinterpreted like the partition
table. Depending on the filesystem and its Linux driver the same wrong
address computations can happen inside the filesystem.

(ISO 9660 has a field to announce the intended logical block size.
But the Linux driver "iso9660" ignores it and assumes 2048 bytes/block.
It is nice enough to convert the LBA values of the filesystem to
512 bytes per logical block addresses if needed. I guess it does the same
for disk devices with 4096 bytes per logical block, but never tested this.)


> Is there any way to make this work?
> Device                            Boot Start       End   Sectors  Size Id Type
> /dev/foo_disk_image                    2048 838858751 838856704  400G 83 Linux

The start address and the number of sectors is integer divisible by 8.
You could change the partition start and size so that the block numbers
address the correct bytes with the 8 fold larger logical block size.

But given that this could still create an unusual situation in the
filesystem driver, i would try to create a device with a logical block
size of 512.
(... or i would create and mount a new filesystem on the new device and
copy the files over from the mounted image file.)


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]


#265935

FromAndy Smith <andy@strugglers.net>
Date2024-01-14 17:20 +0100
Message-ID<HWbnj-34fC-13@gated-at.bofh.it>
In reply to#265927
Hi Thomas,

On Sun, Jan 14, 2024 at 02:33:37PM +0100, Thomas Schmitt wrote:
> I wonder, though, why fdisk would misrepresent the total disk size,
> which i would expect to come from the storage device.
> Maybe fdisk is more confused than i expect.
> 
> Can you show the whole output of fdisk from the new logical device ?

Sorry, I've deleted it now as I've been preparing to try hdparm to
set the underlying HDDs to 512e.

However, what I meant was that fdisk showed a single partition of
3.2TB size, while the entire disk being only the 400G, which is as
you would expect if it thinks the sector count is for 4K sectors,
not 512b ones.

> The start address and the number of sectors is integer divisible by 8.
> You could change the partition start and size so that the block numbers
> address the correct bytes with the 8 fold larger logical block size.
> 
> But given that this could still create an unusual situation in the
> filesystem driver, i would try to create a device with a logical block
> size of 512.

I did try using fdisk on the destination to delete the partition and
recreate it with the correct numbers, but the ext4 driver and fsck
and other filesystem tools were still totally unable to work with
it. Yet, a sha256sum of the disk image file at both sides showed the
same value, so it is the same data.

Is it actually possible to create an LVM volume that thinks it has 512b logical sectors, on an underlying device that is 4Kn?

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]


#265939

From"Thomas Schmitt" <scdbackup@gmx.net>
Date2024-01-14 18:00 +0100
Message-ID<HWc02-34rI-3@gated-at.bofh.it>
In reply to#265935
Hi,

Andy Smith wrote:
> what I meant was that fdisk showed a single partition of
> 3.2TB size, while the entire disk being only the 400G

Then it's what i would expect from fdisk.


> I did try using fdisk on the destination to delete the partition and
> recreate it with the correct numbers, but the ext4 driver and fsck
> and other filesystem tools were still totally unable to work with
> it. Yet, a sha256sum of the disk image file at both sides showed the
> same value, so it is the same data.

I would also expect complaints about read attempts after the end of
the device.


> Is it actually possible to create an LVM volume that thinks it has 512b
> logical sectors, on an underlying device that is 4Kn?

Interesting question.
Googling for LVM instructions ... pvcreate(8), vgcreate(8), lvcreate(8).
Their man pages show no options for selecting block sizes.

losetup(8) has an option which looks helpful:
  -b, --sector-size size
     Set the logical sector size of the loop device in bytes (since Linux
     4.14).
But that would be another software layer between filesystem and physical
device.
(losetup -b 4096 would possibly help to mount an image file from a disk
with logical block size 4K.)


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]


#265945

FromTimothy M Butterworth <timothy.m.butterworth@gmail.com>
Date2024-01-14 19:20 +0100
Message-ID<HWdfr-35lQ-11@gated-at.bofh.it>
In reply to#265915

[Multipart message — attachments visible in raw view] — view raw

On Sun, Jan 14, 2024 at 9:37 AM Andy Smith <andy@strugglers.net> wrote:

> Hi,
>
> I've got a disk image that sits on top of an LVM logical volume
> that is on top of an mdadm RAID-1 that is on top of a pair of:
>
> Device Model:     Samsung SSD 870 EVO 4TB
> Sector Size:      512 bytes logical/physical
>
> so let';s say that is at /dev/foo/disk_image (where /dev/foo is the
> name of the LVM VG and disk_image is the LV)
>
> So,
>
> # fdisk -ul /dev/foo/disk_image
> Disk /dev/foo/disk_image: 400 GiB, 429496729600 bytes, 838860800 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Disk identifier: 0x14409245
>
> Device                            Boot Start       End   Sectors  Size Id
> Type
> /dev/foo_disk_image                    2048 838858751 838856704  400G 83
> Linux
>
> So it's a 400G disk image with an MBR and a single partition, right?
>
> Now, I dd that disk image across the network to another machine
> which has a similar setup, except here the "foo" volume group is on
> a pair of
>
> Device Model:     HGST HUS726T6TALN6L4
> Sector Size:      4096 bytes logical/physical
>

4096 Bytes is 4K sector size.


Now, after the disk_image has arrived, it looks very odd. fdisk
> thinks it is 8 times bigger than it really is, and thinks it has 4K
> sectors. I can't use "kpartx" to get at the partition inside it, and
> fsck.ext4 doesn't like its first partition at all.
>
> Is there any way to make this work?
>
> If necessary and if there is a way, I *can* nuke off the target
> machine's "foo" volume group and recreate the RAID array if I have
> to make it 512e format. But obviously I'd like some way to move this
> disk image and have it still work without having to meddle inside it
> much — it is a VM disk.
>
> Thanks,
> Andy
>
> --
> https://bitfolk.com/ -- No-nonsense VPS hosting
>
>

-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀

[toc] | [prev] | [next] | [standalone]


#265976

FromStefan Monnier <monnier@iro.umontreal.ca>
Date2024-01-15 05:40 +0100
Message-ID<HWmVr-3b5K-1@gated-at.bofh.it>
In reply to#265915
> Now, after the disk_image has arrived, it looks very odd. fdisk
> thinks it is 8 times bigger than it really is, and thinks it has 4K
> sectors. I can't use "kpartx" to get at the partition inside it, and
> fsck.ext4 doesn't like its first partition at all.

Thanks for this experiment.  I was lucky enough not to bump into such
a weird situation and it had never occurred to me that the underlying
block size would "show through" an LV.

> Is there any way to make this work?

[ A quick look at Wikipedia suggests that a GPT partition table won't
  help because that also uses "block" addresses (LBA).  ]

Do you need a partition table?
What happens if you use diskimages that contain directly a filesystem
without going through the trouble of using a partition table?
Does `ext4` also get tripped by the different underlying block size?


        Stefan

[toc] | [prev] | [next] | [standalone]


#265979

FromAndy Smith <andy@strugglers.net>
Date2024-01-15 06:40 +0100
Message-ID<HWnRw-3bI8-5@gated-at.bofh.it>
In reply to#265976
Hi Stefan,

On Sun, Jan 14, 2024 at 11:32:37PM -0500, Stefan Monnier wrote:
> Do you need a partition table?

These are other people's virtual machines so to some extent I don't
have a say on what they put inside them. It is always nice to
understand what is going on though!

> What happens if you use diskimages that contain directly a filesystem
> without going through the trouble of using a partition table?
> Does `ext4` also get tripped by the different underlying block size?

I believe it will also fail but I haven't directly experimented.

On the target host I was able to use fdisk (or gdisk or parted or
whatever…) to change the partition table to be "correct", which
enabled me to then use "kpartx" to expose the partition out of the
disk image as a loop device as usual. However, the ext4 driver and
fsck.ext4 were still unable to find superblocks on this. This
despite a sha256sum of the loop device coming back with the same
hash as a sha256sum of the partition on the source.

So, I believe that the existence or type of a partition table
doesn't matter and similar problems would be encountered if an ext4
filesystem were directly put on the LV. It's just that with there
being a partition table I see problems sooner as the geometry of the
drive is all wrong and tools like kpartx can't work.

Unless there is some way to fiddle this at the LVM PV level, I think
I must try using hdparm to convert the target HDDs to 512 byte
sectors. I will try asking the LVM folks.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]


#266010

FromNicolas George <george@nsup.org>
Date2024-01-15 22:30 +0100
Message-ID<HWCGS-3kRh-15@gated-at.bofh.it>
In reply to#265979

[Multipart message — attachments visible in raw view] — view raw

Nicholas Geovanis (12024-01-15):
> In your dd commands that moved these filesystems, did you specify ibs=
> and/or obs=
> ?
> If so, what values did you use?

Why do you ask this information? How do you think it will be useful?

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]


#266011

FromNicholas Geovanis <nickgeovanis@gmail.com>
Date2024-01-15 22:30 +0100
Message-ID<HWCGS-3kRh-17@gated-at.bofh.it>
In reply to#265979

[Multipart message — attachments visible in raw view] — view raw

On Mon, Jan 15, 2024, 4:58 AM Andy Smith <andy@strugglers.net> wrote:

> On Sun, Jan 14, 2024 at 11:32:37PM -0500, Stefan Monnier wrote:
>
> > What happens if you use diskimages that contain directly a filesystem
> > without going through the trouble of using a partition table?
> > Does `ext4` also get tripped by the different underlying block size?
>
> I believe it will also fail but I haven't directly experimented.
>
> On the target host I was able to use fdisk (or gdisk or parted or
> whatever…) to change the partition table to be "correct", which
> enabled me to then use "kpartx" to expose the partition out of the
> disk image as a loop device as usual. However, the ext4 driver and
> fsck.ext4 were still unable to find superblocks on this. This
> despite a sha256sum of the loop device coming back with the same
> hash as a sha256sum of the partition on the source.
>

In your dd commands that moved these filesystems, did you specify ibs=
and/or obs=
?
If so, what values did you use?
....

> Thanks,
> Andy
>

[toc] | [prev] | [next] | [standalone]


#266028

FromAndy Smith <andy@strugglers.net>
Date2024-01-16 01:40 +0100
Message-ID<HWFEJ-3mEU-7@gated-at.bofh.it>
In reply to#266011
Hello,

On Mon, Jan 15, 2024 at 03:20:51PM -0600, Nicholas Geovanis wrote:
> In your dd commands that moved these filesystems, did you specify ibs=
> and/or obs=

No. I don't see how that would make any difference. I could as well
have used "cat|ssh" instead of "dd|ssh". Also note that the image
files had identical sha256sum on both sides afterwards, so they are
the same data internally. The different behaviour must be triggered
externally to the disk images, which I can only think to be what
Linux considers the correct sector size for the devices.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]


#266081

FromTeemu Likonen <tlikonen@iki.fi>
Date2024-01-16 17:40 +0100
Message-ID<HWUDM-3vTb-19@gated-at.bofh.it>
In reply to#266011

[Multipart message — attachments visible in raw view] — view raw

* 2024-01-15 15:20:51-0600, Nicholas Geovanis wrote:

> In your dd commands that moved these filesystems, did you specify ibs=
> and/or obs=
> ?
> If so, what values did you use?

"dd" is not a special tool for accessing device files. It's a simple
file copy tool: like "cat" or "cp" but with different options. "dd's"
internal buffers can be changed with ibs= and obs= but those have
nothing to do with block device sector size.

(Imagine that "cat" had input buffer size option --ibs= which sets how
many bytes it reads and stores from input file or standard input stream
at one time. The tool could also have --obs= option for setting how many
bytes it writes to the standard output at one time before filling the
output buffer again. That's what "dd's" ibs= and obs= do.)

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 6965F03973F0D4CA22B9410F0F2CAE0E07608462

[toc] | [prev] | [next] | [standalone]


#266104

FromAndy Smith <andy@strugglers.net>
Date2024-01-17 08:30 +0100
Message-ID<HX8x3-3Ewd-3@gated-at.bofh.it>
In reply to#265979
Hi Stefan,

On Mon, Jan 15, 2024 at 05:31:37AM +0000, Andy Smith wrote:
> On Sun, Jan 14, 2024 at 11:32:37PM -0500, Stefan Monnier wrote:
> > What happens if you use diskimages that contain directly a filesystem
> > without going through the trouble of using a partition table?
> > Does `ext4` also get tripped by the different underlying block size?
> 
> I believe it will also fail but I haven't directly experimented.

I've now done a test and putting an ext4 filesystem directly on an
LV and syncing that across results in something that works fine, so
it is an issue with the MBR partition table.

I wonder if there is a safe and reliable way to modify such
partition tables on the destination so as to allow this to work…

(As mentioned, "just don't put a partition table on it" isn't an
ideal solution as these are disk images for virtual machines and
people like to partition those even if it isn't necessary.)

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]


#266337

FromAndy Smith <andy@strugglers.net>
Date2024-01-20 22:30 +0100
Message-ID<HYr4B-4r0s-17@gated-at.bofh.it>
In reply to#265915
On Sun, Jan 14, 2024 at 08:01:52AM +0000, Andy Smith wrote:
> Now, after the disk_image has arrived, it looks very odd. fdisk
> thinks it is 8 times bigger than it really is, and thinks it has 4K
> sectors. I can't use "kpartx" to get at the partition inside it, and
> fsck.ext4 doesn't like its first partition at all.

In case anyone is still following this, I asked about it on the LVM
mailing list and got some more interaction there, but it's still a
mystery.

    https://lore.kernel.org/linux-lvm/9322d044-faff-597d-8cab-bf71ed7e9c74@service4.ru/T/#t

It seems to be off-topic there: I think we've established that
there are problems whether LVM is involved or not, and that LVM has
no function to work around this. But it may continue a little longer
while there are useful suggestions.

Included in that thread is a full list of commands I used to create
a test case on the 512e host and move the disk image to the 4kN
host, in case anyone else wants to replicate the problem. I don't
think you'd need LVM, or MD RAID, to replicate it.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [standalone]


Back to top | Article view | linux.debian.user


csiph-web