Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.user > #265915 > unrolled thread
| Started by | Andy Smith <andy@strugglers.net> |
|---|---|
| First post | 2024-01-14 09:10 +0100 |
| Last post | 2024-01-20 22:30 +0100 |
| Articles | 16 — 9 participants |
Back to article view | Back to linux.debian.user
512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-14 09:10 +0100
Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-14 09:20 +0100
Re: 512e vs 4K sector confusion "Alexander V. Makartsev" <avbetev@gmail.com> - 2024-01-14 12:30 +0100
Re: 512e vs 4K sector confusion Arno Lehmann <al@its-lehmann.de> - 2024-01-14 12:40 +0100
Re: 512e vs 4K sector confusion "Thomas Schmitt" <scdbackup@gmx.net> - 2024-01-14 14:30 +0100
Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-14 17:20 +0100
Re: 512e vs 4K sector confusion "Thomas Schmitt" <scdbackup@gmx.net> - 2024-01-14 18:00 +0100
Re: 512e vs 4K sector confusion Timothy M Butterworth <timothy.m.butterworth@gmail.com> - 2024-01-14 19:20 +0100
Re: 512e vs 4K sector confusion Stefan Monnier <monnier@iro.umontreal.ca> - 2024-01-15 05:40 +0100
Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-15 06:40 +0100
Re: 512e vs 4K sector confusion Nicolas George <george@nsup.org> - 2024-01-15 22:30 +0100
Re: 512e vs 4K sector confusion Nicholas Geovanis <nickgeovanis@gmail.com> - 2024-01-15 22:30 +0100
Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-16 01:40 +0100
Re: 512e vs 4K sector confusion Teemu Likonen <tlikonen@iki.fi> - 2024-01-16 17:40 +0100
Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-17 08:30 +0100
Re: 512e vs 4K sector confusion Andy Smith <andy@strugglers.net> - 2024-01-20 22:30 +0100
| From | Andy Smith <andy@strugglers.net> |
|---|---|
| Date | 2024-01-14 09:10 +0100 |
| Subject | 512e vs 4K sector confusion |
| Message-ID | <HW3J7-2ZIb-5@gated-at.bofh.it> |
Hi, I've got a disk image that sits on top of an LVM logical volume that is on top of an mdadm RAID-1 that is on top of a pair of: Device Model: Samsung SSD 870 EVO 4TB Sector Size: 512 bytes logical/physical so let';s say that is at /dev/foo/disk_image (where /dev/foo is the name of the LVM VG and disk_image is the LV) So, # fdisk -ul /dev/foo/disk_image Disk /dev/foo/disk_image: 400 GiB, 429496729600 bytes, 838860800 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x14409245 Device Boot Start End Sectors Size Id Type /dev/foo_disk_image 2048 838858751 838856704 400G 83 Linux So it's a 400G disk image with an MBR and a single partition, right? Now, I dd that disk image across the network to another machine which has a similar setup, except here the "foo" volume group is on a pair of Device Model: HGST HUS726T6TALN6L4 Sector Size: 4096 bytes logical/physical Now, after the disk_image has arrived, it looks very odd. fdisk thinks it is 8 times bigger than it really is, and thinks it has 4K sectors. I can't use "kpartx" to get at the partition inside it, and fsck.ext4 doesn't like its first partition at all. Is there any way to make this work? If necessary and if there is a way, I *can* nuke off the target machine's "foo" volume group and recreate the RAID array if I have to make it 512e format. But obviously I'd like some way to move this disk image and have it still work without having to meddle inside it much — it is a VM disk. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
[toc] | [next] | [standalone]
| From | Andy Smith <andy@strugglers.net> |
|---|---|
| Date | 2024-01-14 09:20 +0100 |
| Message-ID | <HW3SN-2ZLr-11@gated-at.bofh.it> |
| In reply to | #265915 |
On Sun, Jan 14, 2024 at 08:01:52AM +0000, Andy Smith wrote: > If necessary and if there is a way, I *can* nuke off the target > machine's "foo" volume group and recreate the RAID array if I have > to make it 512e format. But obviously I'd like some way to move this > disk image and have it still work without having to meddle inside it > much — it is a VM disk. I think I may be able to use hdparm to reformat these Ultrastar DCs from 4kn to 512e… hdparm --set-sector-size 512 --please-destroy-my-drive /dev/sdX https://wiki.archlinux.org/title/Advanced_Format#Advanced_Format_hard_disk_drives It is okay to destroy the drive contents as there's nothing working on there yet given this initial failure. I think that is the only way this is going to work without mounting filesystems both sides and using an fs-aware tool like tar|ssh or rsync… which I don't want to do. Thoughts? Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
[toc] | [prev] | [next] | [standalone]
| From | "Alexander V. Makartsev" <avbetev@gmail.com> |
|---|---|
| Date | 2024-01-14 12:30 +0100 |
| Message-ID | <HW6QF-31w6-7@gated-at.bofh.it> |
| In reply to | #265917 |
[Multipart message — attachments visible in raw view] — view raw
On 14.01.2024 13:15, Andy Smith wrote:
> On Sun, Jan 14, 2024 at 08:01:52AM +0000, Andy Smith wrote:
>> If necessary and if there is a way, I *can* nuke off the target
>> machine's "foo" volume group and recreate the RAID array if I have
>> to make it 512e format. But obviously I'd like some way to move this
>> disk image and have it still work without having to meddle inside it
>> much — it is a VM disk.
> I think I may be able to use hdparm to reformat these Ultrastar DCs
> from 4kn to 512e…
>
> hdparm --set-sector-size 512 --please-destroy-my-drive /dev/sdX
>
> https://wiki.archlinux.org/title/Advanced_Format#Advanced_Format_hard_disk_drives
>
> It is okay to destroy the drive contents as there's nothing working
> on there yet given this initial failure.
>
> I think that is the only way this is going to work without mounting
> filesystems both sides and using an fs-aware tool like tar|ssh or
> rsync… which I don't want to do.
>
> Thoughts?
Weird. I can imagine some wild theories. :)
I think it has something to do with a fact that "disk_image" is a LVM
volume and you dd copy "/dev/mapper/VG--name-LV--name" not "/dev/sdc2" .
Could it be that "dd" sees source and destination as block devices and
copies data by sector? I.e. takes 512b sector and place it to the
beginning of 4096b sector and padding the rest 3584b of the sector with
zeroes.
This calls for some tests. I'd create 1G image test file with dos
partition table and a few primary partitions:
$ dd if=/dev/zero of=./1G-disk-image.bin bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1,0 GB, 1000 MiB) copied, 8,07768 s, 130 MB/s
...
$ /sbin/fdisk -l ./1G-disk-image.bin
Disk ./1G-disk-image.bin: 1000 MiB, 1048576000 bytes, 2048000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x143acb59
Device Boot Start End Sectors Size Id Type
./1G-disk-image.bin1 2048 411647 409600 200M 83 Linux
./1G-disk-image.bin2 411648 1435647 1024000 500M 83 Linux
./1G-disk-image.bin3 1435648 2047999 612352 299M 83 Linux
And dd copied it the same way to disk with 4k sectors.
Just to see if it will be inconsistent too, without LVM mapper involved.
I'd also try to clone disk to image file first, with say "partclone"
utility, "scp" the image file and restore it at remote destination with
the same utility after recreating dumped partition table layout using
"sfdisk".
I did this procedure multiple times successfully, but without 4k sector
disks and LVM involvement.
--
With kindest regards, Alexander.
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀
[toc] | [prev] | [next] | [standalone]
| From | Arno Lehmann <al@its-lehmann.de> |
|---|---|
| Date | 2024-01-14 12:40 +0100 |
| Message-ID | <HW70m-31z9-3@gated-at.bofh.it> |
| In reply to | #265917 |
Hi andy, Am 14.01.2024 um 09:15 schrieb Andy Smith: > On Sun, Jan 14, 2024 at 08:01:52AM +0000, Andy Smith wrote: >> If necessary and if there is a way, I *can* nuke off the target >> machine's "foo" volume group and recreate the RAID array if I have >> to make it 512e format. But obviously I'd like some way to move this >> disk image and have it still work without having to meddle inside it >> much — it is a VM disk. > > I think I may be able to use hdparm to reformat these Ultrastar DCs > from 4kn to 512e… ... > Thoughts? Reshaping the target system's storage may be the quickest and most reliable solution. If you expect there may be future cases with similar problems -- the other way around would be much worse, i.e. you have a source storage with 4k sectors, and target that does not support it -- these posts https://unix.stackexchange.com/questions/748208/lvm-and-device-mapper-logical-volume-device-sector-size https://unix.stackexchange.com/questions/676534/creating-an-ssd-partition-with-a-different-block-size might be good starting points to work out a real solution. As far as I know, however, it is not possible to change the block sizes using any of the software layers involved -- they all inherit from their base layer, which means from the actual devices, eventually. Fortunately, I never had to solve such issues myself, but that's probably because I prefer working at the file system level. Block level just has too many pitfalls for my simple tastes ;-) I'm getting curious to see what happens if I have to mix 512- and 4k-sector disks in one RAID, though. Cheers, Arno -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück
[toc] | [prev] | [next] | [standalone]
| From | "Thomas Schmitt" <scdbackup@gmx.net> |
|---|---|
| Date | 2024-01-14 14:30 +0100 |
| Message-ID | <HW8IN-32CB-7@gated-at.bofh.it> |
| In reply to | #265915 |
Hi, Andy Smith wrote: > I've got a disk image that sits on top of an LVM logical volume > that is on top of an mdadm RAID-1 that is on top of a pair of: > Device Model: Samsung SSD 870 EVO 4TB > Sector Size: 512 bytes logical/physical > so let';s say that is at /dev/foo/disk_image (where /dev/foo is the > name of the LVM VG and disk_image is the LV) > So > # fdisk -ul /dev/foo/disk_image > Disk /dev/foo/disk_image: 400 GiB, 429496729600 bytes, 838860800 sectors > Units: sectors of 1 * 512 = 512 bytes fdisk thinks that this device has a logical block size of 512. This information is not stored in MBR partition table (or in GPT) but rather it is a property of the storage device. > Disklabel type: dos > So it's a 400G disk image with an MBR and a single partition, right? Yes. (But nitpicking: It has an MBR partition table with entries other than the protective entry which would indicate the presence of GPT. I.e. GPT contains an MBR too, but its partition table entries are stored outside of the MBR.) > Now, I dd that disk image across the network to another machine > which has a similar setup, except here the "foo" volume group is on > a pair of > Device Model: HGST HUS726T6TALN6L4 > Sector Size: 4096 bytes logical/physical So the logical volume was probably created with the logical block size of the involved physical disk devices. > fdisk > thinks it is 8 times bigger than it really is, and thinks it has 4K > sectors. fdisk asks the device (unless it is a regular file, i guess) for its logical size. Since the partition table gives no hint about its own idea of the bytes/block ratio, you end up with vastely inflated byte numbers. I wonder, though, why fdisk would misrepresent the total disk size, which i would expect to come from the storage device. Maybe fdisk is more confused than i expect. Can you show the whole output of fdisk from the new logical device ? Whatever: Not only the partition table might get misinterpreted like the partition table. Depending on the filesystem and its Linux driver the same wrong address computations can happen inside the filesystem. (ISO 9660 has a field to announce the intended logical block size. But the Linux driver "iso9660" ignores it and assumes 2048 bytes/block. It is nice enough to convert the LBA values of the filesystem to 512 bytes per logical block addresses if needed. I guess it does the same for disk devices with 4096 bytes per logical block, but never tested this.) > Is there any way to make this work? > Device Boot Start End Sectors Size Id Type > /dev/foo_disk_image 2048 838858751 838856704 400G 83 Linux The start address and the number of sectors is integer divisible by 8. You could change the partition start and size so that the block numbers address the correct bytes with the 8 fold larger logical block size. But given that this could still create an unusual situation in the filesystem driver, i would try to create a device with a logical block size of 512. (... or i would create and mount a new filesystem on the new device and copy the files over from the mounted image file.) Have a nice day :) Thomas
[toc] | [prev] | [next] | [standalone]
| From | Andy Smith <andy@strugglers.net> |
|---|---|
| Date | 2024-01-14 17:20 +0100 |
| Message-ID | <HWbnj-34fC-13@gated-at.bofh.it> |
| In reply to | #265927 |
Hi Thomas, On Sun, Jan 14, 2024 at 02:33:37PM +0100, Thomas Schmitt wrote: > I wonder, though, why fdisk would misrepresent the total disk size, > which i would expect to come from the storage device. > Maybe fdisk is more confused than i expect. > > Can you show the whole output of fdisk from the new logical device ? Sorry, I've deleted it now as I've been preparing to try hdparm to set the underlying HDDs to 512e. However, what I meant was that fdisk showed a single partition of 3.2TB size, while the entire disk being only the 400G, which is as you would expect if it thinks the sector count is for 4K sectors, not 512b ones. > The start address and the number of sectors is integer divisible by 8. > You could change the partition start and size so that the block numbers > address the correct bytes with the 8 fold larger logical block size. > > But given that this could still create an unusual situation in the > filesystem driver, i would try to create a device with a logical block > size of 512. I did try using fdisk on the destination to delete the partition and recreate it with the correct numbers, but the ext4 driver and fsck and other filesystem tools were still totally unable to work with it. Yet, a sha256sum of the disk image file at both sides showed the same value, so it is the same data. Is it actually possible to create an LVM volume that thinks it has 512b logical sectors, on an underlying device that is 4Kn? Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
[toc] | [prev] | [next] | [standalone]
| From | "Thomas Schmitt" <scdbackup@gmx.net> |
|---|---|
| Date | 2024-01-14 18:00 +0100 |
| Message-ID | <HWc02-34rI-3@gated-at.bofh.it> |
| In reply to | #265935 |
Hi,
Andy Smith wrote:
> what I meant was that fdisk showed a single partition of
> 3.2TB size, while the entire disk being only the 400G
Then it's what i would expect from fdisk.
> I did try using fdisk on the destination to delete the partition and
> recreate it with the correct numbers, but the ext4 driver and fsck
> and other filesystem tools were still totally unable to work with
> it. Yet, a sha256sum of the disk image file at both sides showed the
> same value, so it is the same data.
I would also expect complaints about read attempts after the end of
the device.
> Is it actually possible to create an LVM volume that thinks it has 512b
> logical sectors, on an underlying device that is 4Kn?
Interesting question.
Googling for LVM instructions ... pvcreate(8), vgcreate(8), lvcreate(8).
Their man pages show no options for selecting block sizes.
losetup(8) has an option which looks helpful:
-b, --sector-size size
Set the logical sector size of the loop device in bytes (since Linux
4.14).
But that would be another software layer between filesystem and physical
device.
(losetup -b 4096 would possibly help to mount an image file from a disk
with logical block size 4K.)
Have a nice day :)
Thomas
[toc] | [prev] | [next] | [standalone]
| From | Timothy M Butterworth <timothy.m.butterworth@gmail.com> |
|---|---|
| Date | 2024-01-14 19:20 +0100 |
| Message-ID | <HWdfr-35lQ-11@gated-at.bofh.it> |
| In reply to | #265915 |
[Multipart message — attachments visible in raw view] — view raw
On Sun, Jan 14, 2024 at 9:37 AM Andy Smith <andy@strugglers.net> wrote: > Hi, > > I've got a disk image that sits on top of an LVM logical volume > that is on top of an mdadm RAID-1 that is on top of a pair of: > > Device Model: Samsung SSD 870 EVO 4TB > Sector Size: 512 bytes logical/physical > > so let';s say that is at /dev/foo/disk_image (where /dev/foo is the > name of the LVM VG and disk_image is the LV) > > So, > > # fdisk -ul /dev/foo/disk_image > Disk /dev/foo/disk_image: 400 GiB, 429496729600 bytes, 838860800 sectors > Units: sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disklabel type: dos > Disk identifier: 0x14409245 > > Device Boot Start End Sectors Size Id > Type > /dev/foo_disk_image 2048 838858751 838856704 400G 83 > Linux > > So it's a 400G disk image with an MBR and a single partition, right? > > Now, I dd that disk image across the network to another machine > which has a similar setup, except here the "foo" volume group is on > a pair of > > Device Model: HGST HUS726T6TALN6L4 > Sector Size: 4096 bytes logical/physical > 4096 Bytes is 4K sector size. Now, after the disk_image has arrived, it looks very odd. fdisk > thinks it is 8 times bigger than it really is, and thinks it has 4K > sectors. I can't use "kpartx" to get at the partition inside it, and > fsck.ext4 doesn't like its first partition at all. > > Is there any way to make this work? > > If necessary and if there is a way, I *can* nuke off the target > machine's "foo" volume group and recreate the RAID array if I have > to make it 512e format. But obviously I'd like some way to move this > disk image and have it still work without having to meddle inside it > much — it is a VM disk. > > Thanks, > Andy > > -- > https://bitfolk.com/ -- No-nonsense VPS hosting > > -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ ⠈⠳⣄⠀⠀
[toc] | [prev] | [next] | [standalone]
| From | Stefan Monnier <monnier@iro.umontreal.ca> |
|---|---|
| Date | 2024-01-15 05:40 +0100 |
| Message-ID | <HWmVr-3b5K-1@gated-at.bofh.it> |
| In reply to | #265915 |
> Now, after the disk_image has arrived, it looks very odd. fdisk
> thinks it is 8 times bigger than it really is, and thinks it has 4K
> sectors. I can't use "kpartx" to get at the partition inside it, and
> fsck.ext4 doesn't like its first partition at all.
Thanks for this experiment. I was lucky enough not to bump into such
a weird situation and it had never occurred to me that the underlying
block size would "show through" an LV.
> Is there any way to make this work?
[ A quick look at Wikipedia suggests that a GPT partition table won't
help because that also uses "block" addresses (LBA). ]
Do you need a partition table?
What happens if you use diskimages that contain directly a filesystem
without going through the trouble of using a partition table?
Does `ext4` also get tripped by the different underlying block size?
Stefan
[toc] | [prev] | [next] | [standalone]
| From | Andy Smith <andy@strugglers.net> |
|---|---|
| Date | 2024-01-15 06:40 +0100 |
| Message-ID | <HWnRw-3bI8-5@gated-at.bofh.it> |
| In reply to | #265976 |
Hi Stefan, On Sun, Jan 14, 2024 at 11:32:37PM -0500, Stefan Monnier wrote: > Do you need a partition table? These are other people's virtual machines so to some extent I don't have a say on what they put inside them. It is always nice to understand what is going on though! > What happens if you use diskimages that contain directly a filesystem > without going through the trouble of using a partition table? > Does `ext4` also get tripped by the different underlying block size? I believe it will also fail but I haven't directly experimented. On the target host I was able to use fdisk (or gdisk or parted or whatever…) to change the partition table to be "correct", which enabled me to then use "kpartx" to expose the partition out of the disk image as a loop device as usual. However, the ext4 driver and fsck.ext4 were still unable to find superblocks on this. This despite a sha256sum of the loop device coming back with the same hash as a sha256sum of the partition on the source. So, I believe that the existence or type of a partition table doesn't matter and similar problems would be encountered if an ext4 filesystem were directly put on the LV. It's just that with there being a partition table I see problems sooner as the geometry of the drive is all wrong and tools like kpartx can't work. Unless there is some way to fiddle this at the LVM PV level, I think I must try using hdparm to convert the target HDDs to 512 byte sectors. I will try asking the LVM folks. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
[toc] | [prev] | [next] | [standalone]
| From | Nicolas George <george@nsup.org> |
|---|---|
| Date | 2024-01-15 22:30 +0100 |
| Message-ID | <HWCGS-3kRh-15@gated-at.bofh.it> |
| In reply to | #265979 |
[Multipart message — attachments visible in raw view] — view raw
Nicholas Geovanis (12024-01-15): > In your dd commands that moved these filesystems, did you specify ibs= > and/or obs= > ? > If so, what values did you use? Why do you ask this information? How do you think it will be useful? -- Nicolas George
[toc] | [prev] | [next] | [standalone]
| From | Nicholas Geovanis <nickgeovanis@gmail.com> |
|---|---|
| Date | 2024-01-15 22:30 +0100 |
| Message-ID | <HWCGS-3kRh-17@gated-at.bofh.it> |
| In reply to | #265979 |
[Multipart message — attachments visible in raw view] — view raw
On Mon, Jan 15, 2024, 4:58 AM Andy Smith <andy@strugglers.net> wrote: > On Sun, Jan 14, 2024 at 11:32:37PM -0500, Stefan Monnier wrote: > > > What happens if you use diskimages that contain directly a filesystem > > without going through the trouble of using a partition table? > > Does `ext4` also get tripped by the different underlying block size? > > I believe it will also fail but I haven't directly experimented. > > On the target host I was able to use fdisk (or gdisk or parted or > whatever…) to change the partition table to be "correct", which > enabled me to then use "kpartx" to expose the partition out of the > disk image as a loop device as usual. However, the ext4 driver and > fsck.ext4 were still unable to find superblocks on this. This > despite a sha256sum of the loop device coming back with the same > hash as a sha256sum of the partition on the source. > In your dd commands that moved these filesystems, did you specify ibs= and/or obs= ? If so, what values did you use? .... > Thanks, > Andy >
[toc] | [prev] | [next] | [standalone]
| From | Andy Smith <andy@strugglers.net> |
|---|---|
| Date | 2024-01-16 01:40 +0100 |
| Message-ID | <HWFEJ-3mEU-7@gated-at.bofh.it> |
| In reply to | #266011 |
Hello, On Mon, Jan 15, 2024 at 03:20:51PM -0600, Nicholas Geovanis wrote: > In your dd commands that moved these filesystems, did you specify ibs= > and/or obs= No. I don't see how that would make any difference. I could as well have used "cat|ssh" instead of "dd|ssh". Also note that the image files had identical sha256sum on both sides afterwards, so they are the same data internally. The different behaviour must be triggered externally to the disk images, which I can only think to be what Linux considers the correct sector size for the devices. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
[toc] | [prev] | [next] | [standalone]
| From | Teemu Likonen <tlikonen@iki.fi> |
|---|---|
| Date | 2024-01-16 17:40 +0100 |
| Message-ID | <HWUDM-3vTb-19@gated-at.bofh.it> |
| In reply to | #266011 |
[Multipart message — attachments visible in raw view] — view raw
* 2024-01-15 15:20:51-0600, Nicholas Geovanis wrote: > In your dd commands that moved these filesystems, did you specify ibs= > and/or obs= > ? > If so, what values did you use? "dd" is not a special tool for accessing device files. It's a simple file copy tool: like "cat" or "cp" but with different options. "dd's" internal buffers can be changed with ibs= and obs= but those have nothing to do with block device sector size. (Imagine that "cat" had input buffer size option --ibs= which sets how many bytes it reads and stores from input file or standard input stream at one time. The tool could also have --obs= option for setting how many bytes it writes to the standard output at one time before filling the output buffer again. That's what "dd's" ibs= and obs= do.) -- /// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/ // OpenPGP: 6965F03973F0D4CA22B9410F0F2CAE0E07608462
[toc] | [prev] | [next] | [standalone]
| From | Andy Smith <andy@strugglers.net> |
|---|---|
| Date | 2024-01-17 08:30 +0100 |
| Message-ID | <HX8x3-3Ewd-3@gated-at.bofh.it> |
| In reply to | #265979 |
Hi Stefan, On Mon, Jan 15, 2024 at 05:31:37AM +0000, Andy Smith wrote: > On Sun, Jan 14, 2024 at 11:32:37PM -0500, Stefan Monnier wrote: > > What happens if you use diskimages that contain directly a filesystem > > without going through the trouble of using a partition table? > > Does `ext4` also get tripped by the different underlying block size? > > I believe it will also fail but I haven't directly experimented. I've now done a test and putting an ext4 filesystem directly on an LV and syncing that across results in something that works fine, so it is an issue with the MBR partition table. I wonder if there is a safe and reliable way to modify such partition tables on the destination so as to allow this to work… (As mentioned, "just don't put a partition table on it" isn't an ideal solution as these are disk images for virtual machines and people like to partition those even if it isn't necessary.) Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
[toc] | [prev] | [next] | [standalone]
| From | Andy Smith <andy@strugglers.net> |
|---|---|
| Date | 2024-01-20 22:30 +0100 |
| Message-ID | <HYr4B-4r0s-17@gated-at.bofh.it> |
| In reply to | #265915 |
On Sun, Jan 14, 2024 at 08:01:52AM +0000, Andy Smith wrote:
> Now, after the disk_image has arrived, it looks very odd. fdisk
> thinks it is 8 times bigger than it really is, and thinks it has 4K
> sectors. I can't use "kpartx" to get at the partition inside it, and
> fsck.ext4 doesn't like its first partition at all.
In case anyone is still following this, I asked about it on the LVM
mailing list and got some more interaction there, but it's still a
mystery.
https://lore.kernel.org/linux-lvm/9322d044-faff-597d-8cab-bf71ed7e9c74@service4.ru/T/#t
It seems to be off-topic there: I think we've established that
there are problems whether LVM is involved or not, and that LVM has
no function to work around this. But it may continue a little longer
while there are useful suggestions.
Included in that thread is a full list of commands I used to create
a test case on the 512e host and move the disk image to the 4kN
host, in case anyone else wants to replicate the problem. I don't
think you'd need LVM, or MD RAID, to replicate it.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
[toc] | [prev] | [standalone]
Back to top | Article view | linux.debian.user
csiph-web