Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1440362 > unrolled thread

Re: [4.7.0rc6] Page Allocation Failures with dm-crypt

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [4.7.0rc6] Page Allocation Failures with dm-crypt Matthias Dahl <ml_linux-kernel@binary-island.eu> - 2016-07-11 10:40 +0200
    Re: [4.7.0rc6] Page Allocation Failures with dm-crypt Mike Snitzer <snitzer@redhat.com> - 2016-07-11 15:20 +0200
      Re: [dm-devel] [4.7.0rc6] Page Allocation Failures with dm-crypt Matthias Dahl <ml_linux-kernel@binary-island.eu> - 2016-07-11 15:30 +0200
        Re: [4.7.0rc6] Page Allocation Failures with dm-crypt Mike Snitzer <snitzer@redhat.com> - 2016-07-11 15:50 +0200
      Re: [4.7.0rc6] Page Allocation Failures with dm-crypt Mike Snitzer <snitzer@redhat.com> - 2016-07-11 15:40 +0200
        Re: [dm-devel] [4.7.0rc6] Page Allocation Failures with dm-crypt Matthias Dahl <ml_linux-kernel@binary-island.eu> - 2016-07-11 16:50 +0200
          Re: [dm-devel] [4.7.0rc6] Page Allocation Failures with dm-crypt Matthias Dahl <ml_linux-kernel@binary-island.eu> - 2016-07-12 10:10 +0200

#1440362 — Re: [4.7.0rc6] Page Allocation Failures with dm-crypt

Hello,

I made a few more tests and here my observations:

- kernels 4.4.8 and 4.5.5 show the same behavior

- the moment dd starts, memory usage spikes rapidly and within a just
   a few seconds has filled up all 32 GiB of RAM

- dd w/ direct i/o works just fine

- mkfs.ext4 unfortunately shows the same behavior as dd w/o direct i/o
   and such makes creation of an ext4 fs on dm-crypt a game of luck

   (much more exposed so with e2fsprogs 1.43.1)

I am kind of puzzled that this bug has seemingly gone so long unnoticed
since it is rather severe and makes dm-crypt unusable to a certain 
degree
for fs encryption (or at least the initial creation of the fs). Am I
missing something here or doing something terribly stupid?

With Kind Regards from Germany
Matthias

-- 
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu
  services: custom software [desktop, mobile, web], server administration

[toc] | [next] | [standalone]

#1440609

On Mon, Jul 11 2016 at  4:31am -0400,
Matthias Dahl <ml_linux-kernel@binary-island.eu> wrote:

> Hello,
> 
> I made a few more tests and here my observations:
> 
> - kernels 4.4.8 and 4.5.5 show the same behavior
> 
> - the moment dd starts, memory usage spikes rapidly and within a just
>   a few seconds has filled up all 32 GiB of RAM
> 
> - dd w/ direct i/o works just fine
> 
> - mkfs.ext4 unfortunately shows the same behavior as dd w/o direct i/o
>   and such makes creation of an ext4 fs on dm-crypt a game of luck
> 
>   (much more exposed so with e2fsprogs 1.43.1)
> 
> I am kind of puzzled that this bug has seemingly gone so long unnoticed
> since it is rather severe and makes dm-crypt unusable to a certain
> degree
> for fs encryption (or at least the initial creation of the fs). Am I
> missing something here or doing something terribly stupid?

Not clear.  Certainly haven't had any reports of memory leaks with
dm-crypt.  Something must explain the execessive nature of your leak but
it isn't a known issue.

Have you tried running with kmemleak enabled?

[toc] | [prev] | [next] | [standalone]

#1440618 — Re: [dm-devel] [4.7.0rc6] Page Allocation Failures with dm-crypt

Hello Mike...

On 2016-07-11 15:18, Mike Snitzer wrote:

> Something must explain the execessive nature of your leak but
> it isn't a known issue.

Since I am currently setting up the new machine, all tests were
performed w/ various live cd images (Fedora Rawhide, Gentoo, ...)
and I saw the exact same behavior everywhere.

> Have you tried running with kmemleak enabled?

I would have to check if that is enabled on the live images but even if
it is, how would that work? The default interval is 10min. If I fire up
a dd, the memory is full within two seconds or so... and after that, the
OOM killer kicks in and all hell breaks loose unfortunately.

I don't think this is a particular unique issue on my side. You could,
if I am right, easily try a Fedora Rawhide image and reproduce it there
yourself. The only unique point here is my RAID10 which is a Intel Rapid
Storage s/w RAID. I have no clue if this could indeed cause such a "bug"
and how.

Thanks,
Matthias

-- 
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu
  services: custom software [desktop, mobile, web], server administration

[toc] | [prev] | [next] | [standalone]

#1440624

On Mon, Jul 11 2016 at  9:27am -0400,
Matthias Dahl <ml_linux-kernel@binary-island.eu> wrote:

> Hello Mike...
> 
> On 2016-07-11 15:18, Mike Snitzer wrote:
> 
> >Something must explain the execessive nature of your leak but
> >it isn't a known issue.
> 
> Since I am currently setting up the new machine, all tests were
> performed w/ various live cd images (Fedora Rawhide, Gentoo, ...)
> and I saw the exact same behavior everywhere.
> 
> >Have you tried running with kmemleak enabled?
> 
> I would have to check if that is enabled on the live images but even if
> it is, how would that work? The default interval is 10min. If I fire up
> a dd, the memory is full within two seconds or so... and after that, the
> OOM killer kicks in and all hell breaks loose unfortunately.

You can control when kmemleak scans.  See Documentation/kmemleak.txt

You could manually trigger a scan just after the dd is started.

But I doubt the livecds have kmemleak compiled into their kernels.

> I don't think this is a particular unique issue on my side. You could,
> if I am right, easily try a Fedora Rawhide image and reproduce it there
> yourself. The only unique point here is my RAID10 which is a Intel Rapid
> Storage s/w RAID. I have no clue if this could indeed cause such a "bug"
> and how.

What is your raid10's full stripesize?  Is your dd IO size of 512K
somehow triggering excess R-M-W cycles which is exacerbating the
problem?

[toc] | [prev] | [next] | [standalone]

#1440621

Inlining reply below after thinking further.

On Mon, Jul 11 2016 at  9:18am -0400,
Mike Snitzer <snitzer@redhat.com> wrote:

> On Mon, Jul 11 2016 at  4:31am -0400,
> Matthias Dahl <ml_linux-kernel@binary-island.eu> wrote:
> 
> > Hello,
> > 
> > I made a few more tests and here my observations:
> > 
> > - kernels 4.4.8 and 4.5.5 show the same behavior
> > 
> > - the moment dd starts, memory usage spikes rapidly and within a just
> >   a few seconds has filled up all 32 GiB of RAM

But that is expected given you're doing an unbounded buffered write to
the device.  What isn't expected, to me anyway, is that the mm subsystem
(or the default knobs for buffered writeback) would be so aggressive
about delaying writeback.

> > - dd w/ direct i/o works just fine

Given that directio works the unbounded buffered IO write you're doing
with dd certainly speaks to that specific area of the mm subsystem.

Why are you doing this test anyway?  Such a large buffered write doesn't
seem to accurately model any application I'm aware of (but obviously it
should still "work").

> > - mkfs.ext4 unfortunately shows the same behavior as dd w/o direct i/o
> >   and such makes creation of an ext4 fs on dm-crypt a game of luck
> > 
> >   (much more exposed so with e2fsprogs 1.43.1)

Now that is weird.  Are you (or the distro you're using) setting any mm
subsystem tunables to really broken values?

> > I am kind of puzzled that this bug has seemingly gone so long unnoticed
> > since it is rather severe and makes dm-crypt unusable to a certain
> > degree
> > for fs encryption (or at least the initial creation of the fs). Am I
> > missing something here or doing something terribly stupid?
> 
> Not clear.  Certainly haven't had any reports of memory leaks with
> dm-crypt.  Something must explain the execessive nature of your leak but
> it isn't a known issue.
> 
> Have you tried running with kmemleak enabled?

I'm now doubting this is a leak...

Mike

[toc] | [prev] | [next] | [standalone]

#1440680 — Re: [dm-devel] [4.7.0rc6] Page Allocation Failures with dm-crypt

Hello Mike...

On 2016-07-11 15:30, Mike Snitzer wrote:

> But that is expected given you're doing an unbounded buffered write to
> the device.  What isn't expected, to me anyway, is that the mm 
> subsystem
> (or the default knobs for buffered writeback) would be so aggressive
> about delaying writeback.

Ok. But, and please correct me if I am wrong, I was under the impression
that only the file caches/buffers were affected, iow, if I use free to
monitor the memory usage, the used memory increases to the point where 
it
consumes all memory, not the buffers/file caches... that is what I am
seeing here.

Also, if I use dd directly on the device w/o dm-crypt in-between, there
is no problem. Sure, buffers increase hugely also... but only those.

> Why are you doing this test anyway?  Such a large buffered write 
> doesn't
> seem to accurately model any application I'm aware of (but obviously it
> should still "work").

It is not a test per se. I simply wanted to fill the partition with 
noise.
And doing it this way is faster than using urandom or anything. ;-) That 
is
why I stumbled over this issue in the first place.

> Now that is weird.  Are you (or the distro you're using) setting any mm
> subsystem tunables to really broken values?

You can see those in my initial mail. I attached the kernel warnings, 
all
sysctl tunables and more. Maybe that helps.

> What is your raid10's full stripesize?

4 disks in RAID10, with a stripe size of 64k.

> Is your dd IO size of 512K somehow triggering excess R-M-W cycles which
> is exacerbating the problem?

The partitions are properly aligned. And as you can see, with that 
stripe
size, there is no issue.

In the meantime I did some further tests: I created an ext2 on the
partition as well as a 60GiB container image on it. I used that image
with dm-crypt, same parameters as before. No matter what I do here, I
cannot trigger the same behavior.

Maybe it is an interaction issue between dm-crypt and the s/w RAID. But
at this point, I have no idea how to further diagnose/test it. If you
can point me in any direction that would be great...

With Kind Regards from Germany
Matthias

-- 
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu
  services: custom software [desktop, mobile, web], server administration

[toc] | [prev] | [next] | [standalone]

#1441124 — Re: [dm-devel] [4.7.0rc6] Page Allocation Failures with dm-crypt

Hello Mike...

It (at least) seems like that the fact that this is a Intel Rapid
Storage RAID10 does play a role after all.

I did several tests this morning, after I finally managed to get
another disk hooked up via USB3 (the best I could do for now).

I tried dm-crypt w/ a single partition and with a Linux s/w RAID1 and
RAID10. Unfortunately the RAID tests were done with several partitions
that were on the same disk since that is all I have for testing.

No matter what I did, I could not get to the point where all memory
got exhausted... not even close.

So with all tests I have done over the last couple of days, the only
one were I could always hit this issue was with my original RAID10
that is a Intel Rapid Storage RAID -- and hooked up to the internal
SATA3 ports instead of the USB3 port (which I hope makes no huge
difference for this issue).

I will post to the linux raid list in a few moments and cc' this list
as well.

Thanks again for the help thus far,
Matthias

-- 
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu
  services: custom software [desktop, mobile, web], server administration

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web