Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1471607 > unrolled thread

Re: DAX can not work on virtual nvdimm device

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: DAX can not work on virtual nvdimm device Xiao Guangrong <guangrong.xiao@linux.intel.com> - 2016-08-29 10:00 +0200
    Re: DAX can not work on virtual nvdimm device Ross Zwisler <ross.zwisler@linux.intel.com> - 2016-08-29 21:40 +0200
      Re: DAX can not work on virtual nvdimm device Xiao Guangrong <guangrong.xiao@linux.intel.com> - 2016-08-30 09:00 +0200
        Re: DAX can not work on virtual nvdimm device Dan Williams <dan.j.williams@intel.com> - 2016-08-30 19:10 +0200
          Re: DAX can not work on virtual nvdimm device Xiao Guangrong <guangrong.xiao@linux.intel.com> - 2016-08-31 11:00 +0200
            Re: DAX can not work on virtual nvdimm device Ross Zwisler <ross.zwisler@linux.intel.com> - 2016-08-31 18:50 +0200
            Re: DAX can not work on virtual nvdimm device Ross Zwisler <ross.zwisler@linux.intel.com> - 2016-09-02 05:00 +0200
              Re: DAX can not work on virtual nvdimm device Jan Kara <jack@suse.cz> - 2016-09-06 17:10 +0200
                Re: DAX can not work on virtual nvdimm device Ross Zwisler <ross.zwisler@linux.intel.com> - 2016-09-08 22:50 +0200
                  Re: DAX can not work on virtual nvdimm device Jan Kara <jack@suse.cz> - 2016-09-09 11:30 +0200

#1471607 — Re: DAX can not work on virtual nvdimm device

Hi Ross,

Sorry for the delay, i just returned back from KVM Forum.

On 08/20/2016 02:30 AM, Ross Zwisler wrote:
> On Fri, Aug 19, 2016 at 07:59:29AM -0700, Dan Williams wrote:
>> On Fri, Aug 19, 2016 at 4:19 AM, Xiao Guangrong
>> <guangrong.xiao@linux.intel.com> wrote:
>>>
>>> Hi Dan,
>>>
>>> Recently, Redhat reported that nvml test suite failed on QEMU/KVM,
>>> more detailed info please refer to:
>>>    https://bugzilla.redhat.com/show_bug.cgi?id=1365721
>>>
>>> The reason for this bug is that the memory region created by mmap()
>>> on the dax-based file was gone so that the region can not be found
>>> in /proc/self/smaps during the runtime.
>>>
>>> This is a simple way to trigger this issue:
>>>    mount -o dax /dev/pmem0 /mnt/pmem/
>>>    vim /mnt/pmem/xxx
>>> then 'vim' is crashed due to segment fault.
>>>
>>> This bug can be reproduced on your tree, the top commit is
>>> 10d7902fa0e82b (dax: unmap/truncate on device shutdown), the kernel
>>> configure file is attached.
>>>
>>> Your thought or comment is highly appreciated.
>>
>> I'm going to be offline until Tuesday, but I will investigate when I'm
>> back.  In the meantime if Ross or Vishal had an opportunity to take a
>> look I wouldn't say "no" :).
>
> I haven't been able to reproduce this vim segfault.  I'm using QEMU v2.6.0,
> and the kernel commit you mentioned, and your kernel config.
>
> Here's my QEMU command line:
>
> sudo ~/qemu/bin/qemu-system-x86_64 /var/lib/libvirt/images/alara.qcow2 \
> -machine pc,nvdimm -m 8G,maxmem=100G,slots=100  -object \
> memory-backend-file,id=mem1,share,mem-path=/dev/pmem0,size=8G -device \
> nvdimm,memdev=mem1,id=nv1 -smp 6 -machine pc,accel=kvm
>
> With this I'm able to mkfs the guest's /dev/pmem0, mount it with -o dax, and
> write a file with vim.

Thanks for your test. That's strange...

>
> Can you reproduce your results with a pmem device created via a memmap kernel
> command line parameter in the guest?  You'll need to update your kernel
> config to enable CONFIG_X86_PMEM_LEGACY and CONFIG_X86_PMEM_LEGACY_DEVICE.
>

Okay, i tested it with mmap=6G!10G, it failed too. So it looks like it's a
filesystem or DAX issue.

More precisely, i figured out the root case that read() returns a wrong value
when it reaches the end of the file, following test case can trigger it:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>

int main(int argc, char *argv[])
{
	char *filename;

	if (argc < 2) {
		printf("arg: filename.\n");
		return -1;
	}

	filename = argv[1];
	printf("test on %s.\n", filename);

	int fd = open(filename, O_RDWR);

	if (fd < 0) {
		perror("open");
		return -1;
	}

	int count = 0;

	while (1) {
		ssize_t ret;
		char buf;

		ret = read(fd, &buf, sizeof(buf));
		if (ret < 0) {
			perror("READ");
			return -1;
		}

		if (ret == 0)
			break;
		if (ret != sizeof(buf)) {
			printf("Count %x Ret %lx sizeof(buf) %lx.\n",
				count, ret, sizeof(buf));
			return -1;
		}

		count++;
		printf("%c", buf);
	}

	printf("\n Good Read.\n");
	return 0;
}



It will fail at "ret != sizeof(buf)", for example, the error output on my
test env is:
       Count 1000 Ret 22f84200 sizeof(buf) 1.

[toc] | [next] | [standalone]

#1472080

On Mon, Aug 29, 2016 at 03:54:10PM +0800, Xiao Guangrong wrote:
> 
> Hi Ross,
> 
> Sorry for the delay, i just returned back from KVM Forum.
> 
> On 08/20/2016 02:30 AM, Ross Zwisler wrote:
> > On Fri, Aug 19, 2016 at 07:59:29AM -0700, Dan Williams wrote:
> > > On Fri, Aug 19, 2016 at 4:19 AM, Xiao Guangrong
> > > <guangrong.xiao@linux.intel.com> wrote:
> > > > 
> > > > Hi Dan,
> > > > 
> > > > Recently, Redhat reported that nvml test suite failed on QEMU/KVM,
> > > > more detailed info please refer to:
> > > >    https://bugzilla.redhat.com/show_bug.cgi?id=1365721
> > > > 
> > > > The reason for this bug is that the memory region created by mmap()
> > > > on the dax-based file was gone so that the region can not be found
> > > > in /proc/self/smaps during the runtime.
> > > > 
> > > > This is a simple way to trigger this issue:
> > > >    mount -o dax /dev/pmem0 /mnt/pmem/
> > > >    vim /mnt/pmem/xxx
> > > > then 'vim' is crashed due to segment fault.
> > > > 
> > > > This bug can be reproduced on your tree, the top commit is
> > > > 10d7902fa0e82b (dax: unmap/truncate on device shutdown), the kernel
> > > > configure file is attached.
> > > > 
> > > > Your thought or comment is highly appreciated.
> > > 
> > > I'm going to be offline until Tuesday, but I will investigate when I'm
> > > back.  In the meantime if Ross or Vishal had an opportunity to take a
> > > look I wouldn't say "no" :).
> > 
> > I haven't been able to reproduce this vim segfault.  I'm using QEMU v2.6.0,
> > and the kernel commit you mentioned, and your kernel config.
> > 
> > Here's my QEMU command line:
> > 
> > sudo ~/qemu/bin/qemu-system-x86_64 /var/lib/libvirt/images/alara.qcow2 \
> > -machine pc,nvdimm -m 8G,maxmem=100G,slots=100  -object \
> > memory-backend-file,id=mem1,share,mem-path=/dev/pmem0,size=8G -device \
> > nvdimm,memdev=mem1,id=nv1 -smp 6 -machine pc,accel=kvm
> > 
> > With this I'm able to mkfs the guest's /dev/pmem0, mount it with -o dax, and
> > write a file with vim.
> 
> Thanks for your test. That's strange...
> 
> > 
> > Can you reproduce your results with a pmem device created via a memmap kernel
> > command line parameter in the guest?  You'll need to update your kernel
> > config to enable CONFIG_X86_PMEM_LEGACY and CONFIG_X86_PMEM_LEGACY_DEVICE.
> > 
> 
> Okay, i tested it with mmap=6G!10G, it failed too. So it looks like it's a
> filesystem or DAX issue.
> 
> More precisely, i figured out the root case that read() returns a wrong value
> when it reaches the end of the file, following test case can trigger it:
> 
> #include <stdlib.h>
> #include <stdio.h>
> #include <unistd.h>
> #include <fcntl.h>
> #include <sys/types.h>
> #include <sys/stat.h>
> 
> int main(int argc, char *argv[])
> {
> 	char *filename;
> 
> 	if (argc < 2) {
> 		printf("arg: filename.\n");
> 		return -1;
> 	}
> 
> 	filename = argv[1];
> 	printf("test on %s.\n", filename);
> 
> 	int fd = open(filename, O_RDWR);
> 
> 	if (fd < 0) {
> 		perror("open");
> 		return -1;
> 	}
> 
> 	int count = 0;
> 
> 	while (1) {
> 		ssize_t ret;
> 		char buf;
> 
> 		ret = read(fd, &buf, sizeof(buf));
> 		if (ret < 0) {
> 			perror("READ");
> 			return -1;
> 		}
> 
> 		if (ret == 0)
> 			break;
> 		if (ret != sizeof(buf)) {
> 			printf("Count %x Ret %lx sizeof(buf) %lx.\n",
> 				count, ret, sizeof(buf));
> 			return -1;
> 		}
> 
> 		count++;
> 		printf("%c", buf);
> 	}
> 
> 	printf("\n Good Read.\n");
> 	return 0;
> }
> 
> 
> 
> It will fail at "ret != sizeof(buf)", for example, the error output on my
> test env is:
>       Count 1000 Ret 22f84200 sizeof(buf) 1.

Can you please verify that you are using "usable" memory for your memmap?  All
the details are here:

https://nvdimm.wiki.kernel.org/how_to_choose_the_correct_memmap_kernel_parameter_for_pmem_on_your_system

My guess is that Boaz was correct, and that your memmap is off using addresses
that don't actually map to memory.

- Ross

[toc] | [prev] | [next] | [standalone]

#1472262

On 08/30/2016 03:30 AM, Ross Zwisler wrote:

>
> Can you please verify that you are using "usable" memory for your memmap?  All
> the details are here:
>
> https://nvdimm.wiki.kernel.org/how_to_choose_the_correct_memmap_kernel_parameter_for_pmem_on_your_system

Sure.

This is the BIOS E820 info in the guest:

e820: BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
BIOS-e820: [mem 0x0000000100000000-0x000000083fffffff] usable
NX (Execute Disable) protection: active
e820: user-defined physical RAM map:
user: [mem 0x0000000000000000-0x000000000009fbff] usable
user: [mem 0x000000000009fc00-0x000000000009ffff] reserved
user: [mem 0x00000000000f0000-0x00000000000fffff] reserved
user: [mem 0x0000000000100000-0x00000000bffdefff] usable
user: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
user: [mem 0x00000000feffc000-0x00000000feffffff] reserved
user: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
user: [mem 0x0000000100000000-0x000000027fffffff] usable
user: [mem 0x0000000280000000-0x00000003ffffffff] persistent (type 12)
user: [mem 0x0000000400000000-0x000000083fffffff] usable

So that the memory we used to emulate PMEM split the 'usable' region.

>
> My guess is that Boaz was correct, and that your memmap is off using addresses
> that don't actually map to memory.

I do not think so. :(

I did mmap-write and mmap-read test, the data written by mmap-write can be
correctly read out, so that the backend memory is really existing.

Thanks!

[toc] | [prev] | [next] | [standalone]

#1472652

On Mon, Aug 29, 2016 at 11:53 PM, Xiao Guangrong
<guangrong.xiao@linux.intel.com> wrote:
>
>
> On 08/30/2016 03:30 AM, Ross Zwisler wrote:
>
>>
>> Can you please verify that you are using "usable" memory for your memmap?
>> All
>> the details are here:
>>
>>
>> https://nvdimm.wiki.kernel.org/how_to_choose_the_correct_memmap_kernel_parameter_for_pmem_on_your_system
>
>
> Sure.
>
> This is the BIOS E820 info in the guest:
>
> e820: BIOS-provided physical RAM map:
> BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
> BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
> BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
> BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
> BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
> BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
> BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
> BIOS-e820: [mem 0x0000000100000000-0x000000083fffffff] usable
> NX (Execute Disable) protection: active
> e820: user-defined physical RAM map:
> user: [mem 0x0000000000000000-0x000000000009fbff] usable
> user: [mem 0x000000000009fc00-0x000000000009ffff] reserved
> user: [mem 0x00000000000f0000-0x00000000000fffff] reserved
> user: [mem 0x0000000000100000-0x00000000bffdefff] usable
> user: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
> user: [mem 0x00000000feffc000-0x00000000feffffff] reserved
> user: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
> user: [mem 0x0000000100000000-0x000000027fffffff] usable
> user: [mem 0x0000000280000000-0x00000003ffffffff] persistent (type 12)
> user: [mem 0x0000000400000000-0x000000083fffffff] usable
>
> So that the memory we used to emulate PMEM split the 'usable' region.
>
>>
>> My guess is that Boaz was correct, and that your memmap is off using
>> addresses
>> that don't actually map to memory.
>
>
> I do not think so. :(
>
> I did mmap-write and mmap-read test, the data written by mmap-write can be
> correctly read out, so that the backend memory is really existing.
>

Can you post your exact reproduction steps?  This test is not failing for me.

[toc] | [prev] | [next] | [standalone]

#1473205

[Multipart message — attachments visible in raw view] — view raw

On 08/31/2016 01:09 AM, Dan Williams wrote:

>
> Can you post your exact reproduction steps?  This test is not failing for me.
>

Sure.

1. make the guest kernel based on your tree, the top commit is
    10d7902fa0e82b (dax: unmap/truncate on device shutdown) and
    the config file can be found in this thread.

2. add guest kernel command line: memmap=6G!10G

3: start the guest:
    x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm --enable-kvm \
    -smp 16 -m 32G,maxmem=100G,slots=100 /other/VMs/centos6.img -monitor stdio

4: in guest:
    mkfs.ext4 /dev/pmem0
    mount -o dax /dev/pmem0  /mnt/pmem/
    echo > /mnt/pmem/xxx
    ./mmap /mnt/pmem/xxx
    ./read /mnt/pmem/xxx

   The source code of mmap and read has been attached in this mail.

   Hopefully, you can detect the error triggered by read test.

Thanks!

[toc] | [prev] | [next] | [standalone]

#1473697

On Wed, Aug 31, 2016 at 04:44:47PM +0800, Xiao Guangrong wrote:
> On 08/31/2016 01:09 AM, Dan Williams wrote:
> > 
> > Can you post your exact reproduction steps?  This test is not failing for me.
> > 
> 
> Sure.
> 
> 1. make the guest kernel based on your tree, the top commit is
>    10d7902fa0e82b (dax: unmap/truncate on device shutdown) and
>    the config file can be found in this thread.
> 
> 2. add guest kernel command line: memmap=6G!10G
> 
> 3: start the guest:
>    x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm --enable-kvm \
>    -smp 16 -m 32G,maxmem=100G,slots=100 /other/VMs/centos6.img -monitor stdio
> 
> 4: in guest:
>    mkfs.ext4 /dev/pmem0
>    mount -o dax /dev/pmem0  /mnt/pmem/
>    echo > /mnt/pmem/xxx
>    ./mmap /mnt/pmem/xxx
>    ./read /mnt/pmem/xxx
> 
>   The source code of mmap and read has been attached in this mail.
> 
>   Hopefully, you can detect the error triggered by read test.
> 
> Thanks!

I'm still unable to reproduce this issue.

I'm using a version of QEMU that I compiled at this commit:

bfc766d (HEAD, tag: v2.6.0) Update version for v2.6.0 release

Here are the options I used for the compile:

./configure --prefix=/home/rzwisler/qemu --target-list=x86_64-softmmu
--enable-kvm --enable-spice --enable-libusb --enable-usb-redir

I used the kernel commit and kernel config you provided.  The mmap is set up
the same, as are the QEMU command line parameters.  

With all this, the tests you provided give the following output:

	# ./mmap /mnt/pmem/xxx
	mmap test on /mnt/pmem/xxx.
	Try to write 0x7f160072d000 for 1000 size.
	Write Done.
	Try to read 0x7f160072d000 for 1000 size.
	Read Done.
	End: 1000.
	Try to fread fd=3 size 1000 sizeof(buf) 1.
	Fread Done.

	# ./read /mnt/pmem/xxx
	test on /mnt/pmem/xxx.
	<snip a bunch of garbage read output>
	 Good Read.

I'm not sure what else to look at.  What do you see in /proc/cpuinfo?  Perhaps
our virtual machine CPUs are advertising different features, and we are going
down different code paths?

Here are my cpuinfo flags in my guest:

flags           : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl eagerfpu pni cx16
x2apic hypervisor lahf_lm

Another thing to do would be to run your test on bare metal on the same
machine and see if you get different results.

Thanks,
- Ross

[toc] | [prev] | [next] | [standalone]

#1474866

On Wed, Aug 31, 2016 at 04:44:47PM +0800, Xiao Guangrong wrote:
> On 08/31/2016 01:09 AM, Dan Williams wrote:
> > 
> > Can you post your exact reproduction steps?  This test is not failing for me.
> > 
> 
> Sure.
> 
> 1. make the guest kernel based on your tree, the top commit is
>    10d7902fa0e82b (dax: unmap/truncate on device shutdown) and
>    the config file can be found in this thread.
> 
> 2. add guest kernel command line: memmap=6G!10G
> 
> 3: start the guest:
>    x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm --enable-kvm \
>    -smp 16 -m 32G,maxmem=100G,slots=100 /other/VMs/centos6.img -monitor stdio
> 
> 4: in guest:
>    mkfs.ext4 /dev/pmem0
>    mount -o dax /dev/pmem0  /mnt/pmem/
>    echo > /mnt/pmem/xxx
>    ./mmap /mnt/pmem/xxx
>    ./read /mnt/pmem/xxx
> 
>   The source code of mmap and read has been attached in this mail.
> 
>   Hopefully, you can detect the error triggered by read test.
> 
> Thanks!

Okay, I think I've isolated this issue.  Xiao's VM was an old CentOS 6 system,
and for some reason ext4+DAX with the old tools found in that VM fails.  I was
able to reproduce this failure with a freshly installed CentOS 6.8 VM.

You can see the failure with his tests, or perhaps more easily with this
series of commands:

  # mkfs.ext4 /dev/pmem0
  # mount -o dax /dev/pmem0  /mnt/pmem/
  # touch /mnt/pmem/x
  # md5sum /mnt/pmem/x
  md5sum: /mnt/pmem/x: Bad address

This sequence of commands works fine in the old CentOS 6 system if you use XFS
instead of ext4, and it works fine with both ext4 and XFS in CentOS 7 and
with recent versions of Fedora.

I've added the ext4 folks to this mail in case they care, but my guess is that
the tools in CentOS 6 are so old that it's not worth worrying about.  For
reference, the kernel in CentOS 6 is based on 2.6.32.  :)  DAX was introduced
in v4.0.

[toc] | [prev] | [next] | [standalone]

#1477495

On Thu 01-09-16 20:57:38, Ross Zwisler wrote:
> On Wed, Aug 31, 2016 at 04:44:47PM +0800, Xiao Guangrong wrote:
> > On 08/31/2016 01:09 AM, Dan Williams wrote:
> > > 
> > > Can you post your exact reproduction steps?  This test is not failing for me.
> > > 
> > 
> > Sure.
> > 
> > 1. make the guest kernel based on your tree, the top commit is
> >    10d7902fa0e82b (dax: unmap/truncate on device shutdown) and
> >    the config file can be found in this thread.
> > 
> > 2. add guest kernel command line: memmap=6G!10G
> > 
> > 3: start the guest:
> >    x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm --enable-kvm \
> >    -smp 16 -m 32G,maxmem=100G,slots=100 /other/VMs/centos6.img -monitor stdio
> > 
> > 4: in guest:
> >    mkfs.ext4 /dev/pmem0
> >    mount -o dax /dev/pmem0  /mnt/pmem/
> >    echo > /mnt/pmem/xxx
> >    ./mmap /mnt/pmem/xxx
> >    ./read /mnt/pmem/xxx
> > 
> >   The source code of mmap and read has been attached in this mail.
> > 
> >   Hopefully, you can detect the error triggered by read test.
> > 
> > Thanks!
> 
> Okay, I think I've isolated this issue.  Xiao's VM was an old CentOS 6 system,
> and for some reason ext4+DAX with the old tools found in that VM fails.  I was
> able to reproduce this failure with a freshly installed CentOS 6.8 VM.
> 
> You can see the failure with his tests, or perhaps more easily with this
> series of commands:
> 
>   # mkfs.ext4 /dev/pmem0
>   # mount -o dax /dev/pmem0  /mnt/pmem/
>   # touch /mnt/pmem/x
>   # md5sum /mnt/pmem/x
>   md5sum: /mnt/pmem/x: Bad address
> 
> This sequence of commands works fine in the old CentOS 6 system if you use XFS
> instead of ext4, and it works fine with both ext4 and XFS in CentOS 7 and
> with recent versions of Fedora.
> 
> I've added the ext4 folks to this mail in case they care, but my guess is that
> the tools in CentOS 6 are so old that it's not worth worrying about.  For
> reference, the kernel in CentOS 6 is based on 2.6.32.  :)  DAX was introduced
> in v4.0.

Hum, can you post 'dumpe2fs -h /dev/pmem0' output from that system when the
md5sum fails? Because the only idea I have is that mkfs.ext4 in CentOS 6
creates the filesystem with a different set of features than more recent
e2fsprogs and so we hit some untested path...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [next] | [standalone]

#1479504

On Tue, Sep 06, 2016 at 05:06:20PM +0200, Jan Kara wrote:
> On Thu 01-09-16 20:57:38, Ross Zwisler wrote:
> > On Wed, Aug 31, 2016 at 04:44:47PM +0800, Xiao Guangrong wrote:
> > > On 08/31/2016 01:09 AM, Dan Williams wrote:
> > > > 
> > > > Can you post your exact reproduction steps?  This test is not failing for me.
> > > > 
> > > 
> > > Sure.
> > > 
> > > 1. make the guest kernel based on your tree, the top commit is
> > >    10d7902fa0e82b (dax: unmap/truncate on device shutdown) and
> > >    the config file can be found in this thread.
> > > 
> > > 2. add guest kernel command line: memmap=6G!10G
> > > 
> > > 3: start the guest:
> > >    x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm --enable-kvm \
> > >    -smp 16 -m 32G,maxmem=100G,slots=100 /other/VMs/centos6.img -monitor stdio
> > > 
> > > 4: in guest:
> > >    mkfs.ext4 /dev/pmem0
> > >    mount -o dax /dev/pmem0  /mnt/pmem/
> > >    echo > /mnt/pmem/xxx
> > >    ./mmap /mnt/pmem/xxx
> > >    ./read /mnt/pmem/xxx
> > > 
> > >   The source code of mmap and read has been attached in this mail.
> > > 
> > >   Hopefully, you can detect the error triggered by read test.
> > > 
> > > Thanks!
> > 
> > Okay, I think I've isolated this issue.  Xiao's VM was an old CentOS 6 system,
> > and for some reason ext4+DAX with the old tools found in that VM fails.  I was
> > able to reproduce this failure with a freshly installed CentOS 6.8 VM.
> > 
> > You can see the failure with his tests, or perhaps more easily with this
> > series of commands:
> > 
> >   # mkfs.ext4 /dev/pmem0
> >   # mount -o dax /dev/pmem0  /mnt/pmem/
> >   # touch /mnt/pmem/x
> >   # md5sum /mnt/pmem/x
> >   md5sum: /mnt/pmem/x: Bad address
> > 
> > This sequence of commands works fine in the old CentOS 6 system if you use XFS
> > instead of ext4, and it works fine with both ext4 and XFS in CentOS 7 and
> > with recent versions of Fedora.
> > 
> > I've added the ext4 folks to this mail in case they care, but my guess is that
> > the tools in CentOS 6 are so old that it's not worth worrying about.  For
> > reference, the kernel in CentOS 6 is based on 2.6.32.  :)  DAX was introduced
> > in v4.0.
> 
> Hum, can you post 'dumpe2fs -h /dev/pmem0' output from that system when the
> md5sum fails? Because the only idea I have is that mkfs.ext4 in CentOS 6
> creates the filesystem with a different set of features than more recent
> e2fsprogs and so we hit some untested path...

Sure, here's the output:

# dumpe2fs -h /dev/pmem0 
dumpe2fs 1.41.12 (17-May-2010)
Filesystem volume name:   <none>
Last mounted on:          /mnt/pmem
Filesystem UUID:          4cd8a836-cc54-4c59-ae0a-4a26bab0f8bc
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype
needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg
dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1048576
Block count:              4194304
Reserved block count:     209715
Free blocks:              4084463
Free inodes:              1048565
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1023
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
RAID stride:              1
Flex block group size:    16
Filesystem created:       Thu Sep  8 14:45:31 2016
Last mount time:          Thu Sep  8 14:45:39 2016
Last write time:          Thu Sep  8 14:45:39 2016
Mount count:              1
Maximum mount count:      21
Last checked:             Thu Sep  8 14:45:31 2016
Check interval:           15552000 (6 months)
Next check after:         Tue Mar  7 13:45:31 2017
Lifetime writes:          388 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      19cad581-c46a-4212-bfa0-d527ff55db49
Journal backup:           inode blocks
Journal features:         (none)
Journal size:             128M
Journal length:           32768
Journal sequence:         0x00000002
Journal start:            1

[toc] | [prev] | [next] | [standalone]

#1479789

On Thu 08-09-16 14:47:08, Ross Zwisler wrote:
> On Tue, Sep 06, 2016 at 05:06:20PM +0200, Jan Kara wrote:
> > On Thu 01-09-16 20:57:38, Ross Zwisler wrote:
> > > On Wed, Aug 31, 2016 at 04:44:47PM +0800, Xiao Guangrong wrote:
> > > > On 08/31/2016 01:09 AM, Dan Williams wrote:
> > > > > 
> > > > > Can you post your exact reproduction steps?  This test is not failing for me.
> > > > > 
> > > > 
> > > > Sure.
> > > > 
> > > > 1. make the guest kernel based on your tree, the top commit is
> > > >    10d7902fa0e82b (dax: unmap/truncate on device shutdown) and
> > > >    the config file can be found in this thread.
> > > > 
> > > > 2. add guest kernel command line: memmap=6G!10G
> > > > 
> > > > 3: start the guest:
> > > >    x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm --enable-kvm \
> > > >    -smp 16 -m 32G,maxmem=100G,slots=100 /other/VMs/centos6.img -monitor stdio
> > > > 
> > > > 4: in guest:
> > > >    mkfs.ext4 /dev/pmem0
> > > >    mount -o dax /dev/pmem0  /mnt/pmem/
> > > >    echo > /mnt/pmem/xxx
> > > >    ./mmap /mnt/pmem/xxx
> > > >    ./read /mnt/pmem/xxx
> > > > 
> > > >   The source code of mmap and read has been attached in this mail.
> > > > 
> > > >   Hopefully, you can detect the error triggered by read test.
> > > > 
> > > > Thanks!
> > > 
> > > Okay, I think I've isolated this issue.  Xiao's VM was an old CentOS 6 system,
> > > and for some reason ext4+DAX with the old tools found in that VM fails.  I was
> > > able to reproduce this failure with a freshly installed CentOS 6.8 VM.
> > > 
> > > You can see the failure with his tests, or perhaps more easily with this
> > > series of commands:
> > > 
> > >   # mkfs.ext4 /dev/pmem0
> > >   # mount -o dax /dev/pmem0  /mnt/pmem/
> > >   # touch /mnt/pmem/x
> > >   # md5sum /mnt/pmem/x
> > >   md5sum: /mnt/pmem/x: Bad address
> > > 
> > > This sequence of commands works fine in the old CentOS 6 system if you use XFS
> > > instead of ext4, and it works fine with both ext4 and XFS in CentOS 7 and
> > > with recent versions of Fedora.
> > > 
> > > I've added the ext4 folks to this mail in case they care, but my guess is that
> > > the tools in CentOS 6 are so old that it's not worth worrying about.  For
> > > reference, the kernel in CentOS 6 is based on 2.6.32.  :)  DAX was introduced
> > > in v4.0.
> > 
> > Hum, can you post 'dumpe2fs -h /dev/pmem0' output from that system when the
> > md5sum fails? Because the only idea I have is that mkfs.ext4 in CentOS 6
> > creates the filesystem with a different set of features than more recent
> > e2fsprogs and so we hit some untested path...
> 
> Sure, here's the output:
> 
> # dumpe2fs -h /dev/pmem0 
> dumpe2fs 1.41.12 (17-May-2010)
> Filesystem volume name:   <none>
> Last mounted on:          /mnt/pmem
> Filesystem UUID:          4cd8a836-cc54-4c59-ae0a-4a26bab0f8bc
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic)
> Filesystem features:      has_journal ext_attr resize_inode dir_index filetype
> needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg
> dir_nlink extra_isize
> Filesystem flags:         signed_directory_hash 
> Default mount options:    (none)
> Filesystem state:         clean
> Errors behavior:          Continue
> Filesystem OS type:       Linux
> Inode count:              1048576
> Block count:              4194304
> Reserved block count:     209715
> Free blocks:              4084463
> Free inodes:              1048565
> First block:              0
> Block size:               4096
> Fragment size:            4096
> Reserved GDT blocks:      1023
> Blocks per group:         32768
> Fragments per group:      32768
> Inodes per group:         8192
> Inode blocks per group:   512
> RAID stride:              1
> Flex block group size:    16
> Filesystem created:       Thu Sep  8 14:45:31 2016
> Last mount time:          Thu Sep  8 14:45:39 2016
> Last write time:          Thu Sep  8 14:45:39 2016
> Mount count:              1
> Maximum mount count:      21
> Last checked:             Thu Sep  8 14:45:31 2016
> Check interval:           15552000 (6 months)
> Next check after:         Tue Mar  7 13:45:31 2017
> Lifetime writes:          388 MB
> Reserved blocks uid:      0 (user root)
> Reserved blocks gid:      0 (group root)
> First inode:              11
> Inode size:	          256
> Required extra isize:     28
> Desired extra isize:      28
> Journal inode:            8
> Default directory hash:   half_md4
> Directory Hash Seed:      19cad581-c46a-4212-bfa0-d527ff55db49
> Journal backup:           inode blocks
> Journal features:         (none)
> Journal size:             128M
> Journal length:           32768
> Journal sequence:         0x00000002
> Journal start:            1

Hum, nothing unusual in there. I've tried reproducing on a local SLE11 SP3
machine (which is from about the same time) but everything works as
expected there. Shrug...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web