Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1633770 > unrolled thread

[PATCH RFC] hugetlbfs 'noautofill' mount option

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH RFC] hugetlbfs 'noautofill' mount option Prakash Sangappa <prakash.sangappa@oracle.com> - 2017-05-01 20:10 +0200
    Re: [PATCH RFC] hugetlbfs 'noautofill' mount option Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-05-02 13:00 +0200
      Re: [PATCH RFC] hugetlbfs 'noautofill' mount option Prakash Sangappa <prakash.sangappa@oracle.com> - 2017-05-02 18:10 +0200
    Re: [PATCH RFC] hugetlbfs 'noautofill' mount option Dave Hansen <dave.hansen@intel.com> - 2017-05-02 23:40 +0200
      Re: [PATCH RFC] hugetlbfs 'noautofill' mount option Prakash Sangappa <prakash.sangappa@oracle.com> - 2017-05-03 01:40 +0200
        Re: [PATCH RFC] hugetlbfs 'noautofill' mount option Dave Hansen <dave.hansen@intel.com> - 2017-05-03 01:50 +0200
          Re: [PATCH RFC] hugetlbfs 'noautofill' mount option Prakash Sangappa <prakash.sangappa@oracle.com> - 2017-05-03 21:10 +0200
            Re: [PATCH RFC] hugetlbfs 'noautofill' mount option Prakash Sangappa <prakash.sangappa@oracle.com> - 2017-05-08 08:00 +0200
            Re: [PATCH RFC] hugetlbfs 'noautofill' mount option Dave Hansen <dave.hansen@intel.com> - 2017-05-08 18:00 +0200
              Re: [PATCH RFC] hugetlbfs 'noautofill' mount option "prakash.sangappa" <prakash.sangappa@oracle.com> - 2017-05-09 00:20 +0200
                Re: [PATCH RFC] hugetlbfs 'noautofill' mount option Christoph Hellwig <hch@infradead.org> - 2017-05-09 11:00 +0200
                  Re: [PATCH RFC] hugetlbfs 'noautofill' mount option Prakash Sangappa <prakash.sangappa@oracle.com> - 2017-05-09 23:10 +0200

#1633770 — [PATCH RFC] hugetlbfs 'noautofill' mount option

Some applications like a database use hugetblfs for performance
reasons. Files on hugetlbfs filesystem are created and huge pages
allocated using fallocate() API. Pages are deallocated/freed using
fallocate() hole punching support that has been added to hugetlbfs.
These files are mmapped and accessed by many processes as shared memory.
Such applications keep track of which offsets in the hugetlbfs file have
pages allocated.

Any access to mapped address over holes in the file, which can occur due
to bugs in the application, is considered invalid and expect the process
to simply receive a SIGBUS.  However, currently when a hole in the file is
accessed via the mapped address, kernel/mm attempts to automatically
allocate a page at page fault time, resulting in implicitly filling the hole
in the file. This may not be the desired behavior for applications like the
database that want to explicitly manage page allocations of hugetlbfs files.

This patch adds a new hugetlbfs mount option 'noautofill', to indicate that
pages should not be allocated at page fault time when accessed thru mmapped
address.

Signed-off-by: Prakash <prakash.sangappa@oracle.com>
---
fs/hugetlbfs/inode.c    | 11 ++++++++++-
  include/linux/hugetlb.h |  1 +
  mm/hugetlb.c            |  5 +++++
  3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 8f96461..8342ee9 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -53,6 +53,7 @@ struct hugetlbfs_config {
      long    nr_inodes;
      struct hstate *hstate;
      long    min_hpages;
+    long    noautofill;
  };

  struct hugetlbfs_inode_info {
@@ -71,7 +72,7 @@ enum {
      Opt_size, Opt_nr_inodes,
      Opt_mode, Opt_uid, Opt_gid,
      Opt_pagesize, Opt_min_size,
-    Opt_err,
+    Opt_noautofill, Opt_err,
  };

  static const match_table_t tokens = {
@@ -82,6 +83,7 @@ static const match_table_t tokens = {
      {Opt_gid,    "gid=%u"},
      {Opt_pagesize,    "pagesize=%s"},
      {Opt_min_size,    "min_size=%s"},
+    {Opt_noautofill,    "noautofill"},
      {Opt_err,    NULL},
  };

@@ -1109,6 +1111,11 @@ hugetlbfs_parse_options(char *options, struct
hugetlbfs_config *pconfig)
              break;
          }

+        case Opt_noautofill: {
+            pconfig->noautofill = 1;
+            break;
+        }
+
          default:
              pr_err("Bad mount option: \"%s\"\n", p);
              return -EINVAL;
@@ -1157,6 +1164,7 @@ hugetlbfs_fill_super(struct super_block *sb, void
*data, int silent)
      config.mode = 0755;
      config.hstate = &default_hstate;
      config.min_hpages = -1; /* No default minimum size */
+    config.noautofill = 0;
      ret = hugetlbfs_parse_options(data, &config);
      if (ret)
          return ret;
@@ -1170,6 +1178,7 @@ hugetlbfs_fill_super(struct super_block *sb, void
*data, int silent)
      sbinfo->max_inodes = config.nr_inodes;
      sbinfo->free_inodes = config.nr_inodes;
      sbinfo->spool = NULL;
+    sbinfo->noautofill = config.noautofill;
      /*
       * Allocate and initialize subpool if maximum or minimum size is
       * specified.  Any needed reservations (for minimim size) are taken
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 503099d..2f37e0c 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -259,6 +259,7 @@ struct hugetlbfs_sb_info {
      spinlock_t    stat_lock;
      struct hstate *hstate;
      struct hugepage_subpool *spool;
+    int    noautofill; /* don't allocate page to fill hole at fault time */
  };

  static inline struct hugetlbfs_sb_info *HUGETLBFS_SB(struct
super_block *sb)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index a7aa811..11655ef 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3715,6 +3715,11 @@ static int hugetlb_no_page(struct mm_struct *mm,
struct vm_area_struct *vma,
              goto out;
          }

+        if (HUGETLBFS_SB(mapping->host->i_sb)->noautofill) {
+            ret = VM_FAULT_SIGBUS;
+            goto out;
+        }
+
          page = alloc_huge_page(vma, address, 0);
          if (IS_ERR(page)) {
              ret = PTR_ERR(page);
-- 
2.7.4

[toc] | [next] | [standalone]

#1634337

On 05/01/2017 11:30 PM, Prakash Sangappa wrote:
> Some applications like a database use hugetblfs for performance
> reasons. Files on hugetlbfs filesystem are created and huge pages
> allocated using fallocate() API. Pages are deallocated/freed using
> fallocate() hole punching support that has been added to hugetlbfs.
> These files are mmapped and accessed by many processes as shared memory.
> Such applications keep track of which offsets in the hugetlbfs file have
> pages allocated.
> 
> Any access to mapped address over holes in the file, which can occur due

s/mapped/unmapped/ ^ ?

> to bugs in the application, is considered invalid and expect the process
> to simply receive a SIGBUS.  However, currently when a hole in the file is
> accessed via the mapped address, kernel/mm attempts to automatically
> allocate a page at page fault time, resulting in implicitly filling the
> hole

But this is expected when you try to control the file allocation from
a mapped address. Any changes while walking past or writing the range
in the memory mapped should reflect exactly in the file on the disk.
Why its not a valid behavior ?

> in the file. This may not be the desired behavior for applications like the
> database that want to explicitly manage page allocations of hugetlbfs
> files.
> 
> This patch adds a new hugetlbfs mount option 'noautofill', to indicate that
> pages should not be allocated at page fault time when accessed thru mmapped
> address.

When the page should be allocated for mapping ?

[toc] | [prev] | [next] | [standalone]

#1634497

On 5/2/17 3:53 AM, Anshuman Khandual wrote:
> On 05/01/2017 11:30 PM, Prakash Sangappa wrote:
>> Some applications like a database use hugetblfs for performance
>> reasons. Files on hugetlbfs filesystem are created and huge pages
>> allocated using fallocate() API. Pages are deallocated/freed using
>> fallocate() hole punching support that has been added to hugetlbfs.
>> These files are mmapped and accessed by many processes as shared memory.
>> Such applications keep track of which offsets in the hugetlbfs file have
>> pages allocated.
>>
>> Any access to mapped address over holes in the file, which can occur due
> s/mapped/unmapped/ ^ ?

It is 'mapped' address.

>
>> to bugs in the application, is considered invalid and expect the process
>> to simply receive a SIGBUS.  However, currently when a hole in the file is
>> accessed via the mapped address, kernel/mm attempts to automatically
>> allocate a page at page fault time, resulting in implicitly filling the
>> hole
> But this is expected when you try to control the file allocation from
> a mapped address. Any changes while walking past or writing the range
> in the memory mapped should reflect exactly in the file on the disk.
> Why its not a valid behavior ?
Sure, that is a valid behavior. However, hugetlbfs is a pesudo filesystem
and the purpose is for applications to use hugepage memory. The contents
of these filesystem are not backed by disk nor are they swapped out.

The proposed new behavior is only applicable for hugetlbfs filesystem 
mounted
with the new 'noautofill' mount option. The file's page allocation/free 
are managed
using the 'fallocate()' API.

For hugetlbfs filesystems mounted without this option, there is no 
change in behavior.

>> in the file. This may not be the desired behavior for applications like the
>> database that want to explicitly manage page allocations of hugetlbfs
>> files.
>>
>> This patch adds a new hugetlbfs mount option 'noautofill', to indicate that
>> pages should not be allocated at page fault time when accessed thru mmapped
>> address.
> When the page should be allocated for mapping ?
The application would allocate/free file pages using the fallocate() API.

[toc] | [prev] | [next] | [standalone]

#1634632

On 05/01/2017 11:00 AM, Prakash Sangappa wrote:
> This patch adds a new hugetlbfs mount option 'noautofill', to indicate that
> pages should not be allocated at page fault time when accessed thru mmapped
> address.

I think the main argument against doing something like this is further
specializing hugetlbfs.  I was really hoping that userfaultfd would be
usable for your needs here.

Could you elaborate on other options that you considered?  Did you look
at userfaultfd?  What about an madvise() option that disallows backing
allocations?

[toc] | [prev] | [next] | [standalone]

#1634670

On 5/2/17 2:32 PM, Dave Hansen wrote:
> On 05/01/2017 11:00 AM, Prakash Sangappa wrote:
>> This patch adds a new hugetlbfs mount option 'noautofill', to indicate that
>> pages should not be allocated at page fault time when accessed thru mmapped
>> address.
> I think the main argument against doing something like this is further
> specializing hugetlbfs.  I was really hoping that userfaultfd would be
> usable for your needs here.
>
> Could you elaborate on other options that you considered?  Did you look
> at userfaultfd?  What about an madvise() option that disallows backing
> allocations?


Yes, we did consider userfaultfd and madvise(). The use case in mind is 
the database.

With a database, large number of single threaded processes are involved 
which
will map hugetlbfs file and use it for shared memory. The concern with 
using
userfaultfd is the overhead of setup and having an additional thread per 
process
to monitor the userfaultfd. Even if the additional thread can be 
avoided, by using
an external monitor process and  each process sending the userfaultfd to 
this
monitor process, setup overhead exists.

Similarly, a madvise() option also requires additional system call by every
process mapping the file, this is considered a overhead for the database.

If we do consider a new madvise() option, will it be acceptable since 
this will be
specifically for hugetlbfs file mappings? If so, would a new flag to mmap()
call itself be acceptable, which would define the proposed behavior?. 
That way
no additional system calls need to be made. Again this mmap flag would be
applicable  specifically to hugetlbfs file mappings

With the proposed mount option, it would enforce one consistent behavior
and the application using this filesystem would not have to take additional
steps as with userfaultfd or madvise().

[toc] | [prev] | [next] | [standalone]

#1634672

On 05/02/2017 04:34 PM, Prakash Sangappa wrote:
> Similarly, a madvise() option also requires additional system call by every
> process mapping the file, this is considered a overhead for the database.

How long-lived are these processes?  For a database, I'd assume that
this would happen a single time, or a single time per mmap() at process
startup time.  Such a syscall would be doing something on the order of
taking mmap_sem, walking the VMA tree, setting a bit per VMA, and
unlocking.  That's a pretty cheap one-time cost...

> If we do consider a new madvise() option, will it be acceptable
> since this will be specifically for hugetlbfs file mappings?

Ideally, it would be something that is *not* specifically for hugetlbfs.
 MADV_NOAUTOFILL, for instance, could be defined to SIGSEGV whenever
memory is touched that was not populated with MADV_WILLNEED, mlock(), etc...

> If so,
> would a new flag to mmap() call itself be acceptable, which would
> define the proposed behavior?. That way no additional system calls
> need to be made.

I don't feel super strongly about it, but I guess an mmap() flag could
work too.

[toc] | [prev] | [next] | [standalone]

#1635222

On 5/2/17 4:43 PM, Dave Hansen wrote:

> On 05/02/2017 04:34 PM, Prakash Sangappa wrote:
>> Similarly, a madvise() option also requires additional system call by every
>> process mapping the file, this is considered a overhead for the database.
> How long-lived are these processes?  For a database, I'd assume that
> this would happen a single time, or a single time per mmap() at process
> startup time.  Such a syscall would be doing something on the order of
> taking mmap_sem, walking the VMA tree, setting a bit per VMA, and
> unlocking.  That's a pretty cheap one-time cost...
Plus a call into the filesystem (a_ops?) to check if the underlying 
filesystem
supports not filling holes to mapped access before setting the bit per vma.
Although the overhead may not be that bad.

Database processes can exit and new once started, for instance, depending on
database activity.


>> If we do consider a new madvise() option, will it be acceptable
>> since this will be specifically for hugetlbfs file mappings?
> Ideally, it would be something that is *not* specifically for hugetlbfs.
>   MADV_NOAUTOFILL, for instance, could be defined to SIGSEGV whenever
> memory is touched that was not populated with MADV_WILLNEED, mlock(), etc...

If this is a generic advice type, necessary support will have to be 
implemented
in various filesystems which can support this.

The proposed behavior for 'noautofill' was to not fill holes in 
files(like sparse files).
In the page fault path, mm would not know if the mmapped address on which
the fault occurred, is over a hole in the file or just that the page is 
not available
in the page cache. The underlying filesystem would be called and it 
determines
if it is a hole and that is where it would fail and not fill the hole, 
if this support is added.
Normally, filesystem which support sparse files(holes in file) 
automatically fill the hole
when accessed. Then there is the issue of file system block size and 
page size. If the
block sizes are smaller then page size, it could mean the noautofill 
would only work
if the hole size is equal to  or a multiple of, page size?

In case of hugetlbfs it is much straight forward. Since this filesystem 
is not like a normal
filesystems and and the file sizes are multiple of huge pages. The hole 
will be a multiple
of the huge page size. For this reason then should the advise be 
specific to hugetlbfs?


>
>> If so,
>> would a new flag to mmap() call itself be acceptable, which would
>> define the proposed behavior?. That way no additional system calls
>> need to be made.
> I don't feel super strongly about it, but I guess an mmap() flag could
> work too.
>

Same goes with the mmap call, if it is a generic flag.

-Prakash.

[toc] | [prev] | [next] | [standalone]

#1637221

On 5/3/17 12:02 PM, Prakash Sangappa wrote:
> On 5/2/17 4:43 PM, Dave Hansen wrote:
>
>> Ideally, it would be something that is *not* specifically for hugetlbfs.
>>   MADV_NOAUTOFILL, for instance, could be defined to SIGSEGV whenever
>> memory is touched that was not populated with MADV_WILLNEED, mlock(), 
>> etc...
>
> If this is a generic advice type, necessary support will have to be 
> implemented
> in various filesystems which can support this.
>
> The proposed behavior for 'noautofill' was to not fill holes in 
> files(like sparse files).
> In the page fault path, mm would not know if the mmapped address on which
> the fault occurred, is over a hole in the file or just that the page 
> is not available
> in the page cache. The underlying filesystem would be called and it 
> determines
> if it is a hole and that is where it would fail and not fill the hole, 
> if this support is added.
> Normally, filesystem which support sparse files(holes in file) 
> automatically fill the hole
> when accessed. Then there is the issue of file system block size and 
> page size. If the
> block sizes are smaller then page size, it could mean the noautofill 
> would only work
> if the hole size is equal to  or a multiple of, page size?
>
> In case of hugetlbfs it is much straight forward. Since this 
> filesystem is not like a normal
> filesystems and and the file sizes are multiple of huge pages. The 
> hole will be a multiple
> of the huge page size. For this reason then should the advise be 
> specific to hugetlbfs?
>
>


Any further comments? I think introducing a general madvise option or a 
mmap flag applicable to all filesystems, may not be required. The 
'noautofill' behavior would be specifically useful in hugetlbfs filesystem.

So, if it is specific to hugetlbfs, will the mount option be ok? 
Otherwise adding a madvise / mmap option specific to hugetlbfs, be 
preferred?

[toc] | [prev] | [next] | [standalone]

#1637536

On 05/03/2017 12:02 PM, Prakash Sangappa wrote:
>>> If we do consider a new madvise()option, will it be acceptable
>>> since this will be specifically for hugetlbfs file mappings?
>> Ideally, it would be something that is *not* specifically for
>> hugetlbfs. MADV_NOAUTOFILL, for instance, could be defined to
>> SIGSEGV whenever memory is touched that was not populated with
>> MADV_WILLNEED, mlock(), etc...
> 
> If this is a generic advice type, necessary support will have to be 
> implemented in various filesystems which can support this.

Yep.

> The proposed behavior for 'noautofill' was to not fill holes in 
> files(like sparse files). In the page fault path, mm would not know
> if the mmapped address on which the fault occurred, is over a hole in
> the file or just that the page is not available in the page cache.

It depends on how you define the feature.  I think you have three choices:

1. "Error" on page fault.  Require all access to be pre-faulted.
2. Allow faults, but "Error" if page cache has to be allocated
3. Allow faults and page cache allocations, but error on filesystem
   backing storage allocation.

All of those are useful in some cases.  But the implementations probably
happen in different places:

#1 can be implemented in core mm code
#2 can be implemented in the VFS
#3 needs filesystem involvement

> The underlying filesystem would be called and it determines if it is
> a hole and that is where it would fail and not fill the hole, if this
> support is added. Normally, filesystem which support sparse
> files(holes in file) automatically fill the hole when accessed. Then
> there is the issue of file system block size and page size. If the 
> block sizes are smaller then page size, it could mean the noautofill 
> would only work if the hole size is equal to or a multiple of, page
> size?

It depends on how you define the feature whether this is true.

> In case of hugetlbfs it is much straight forward. Since this
> filesystem is not like a normal filesystems and and the file sizes
> are multiple of huge pages. The hole will be a multiple of the huge
> page size. For this reason then should the advise be specific to
> hugetlbfs?

Let me paraphrase: it's simpler to implement it if it's specific to
hugetlbfs, thus we should implement it only for hugetlbfs, and keep it
specific to hugetlbfs.

The bigger question is: do we want to continue adding to the complexity
of hugetlbfs and increase its divergence from the core mm?

[toc] | [prev] | [next] | [standalone]

#1637764

On 05/08/2017 08:58 AM, Dave Hansen wrote:
>
> It depends on how you define the feature.  I think you have three choices:
>
> 1. "Error" on page fault.  Require all access to be pre-faulted.
> 2. Allow faults, but "Error" if page cache has to be allocated
> 3. Allow faults and page cache allocations, but error on filesystem
>     backing storage allocation.
>
> All of those are useful in some cases.  But the implementations probably
> happen in different places:
>
> #1 can be implemented in core mm code
> #2 can be implemented in the VFS
> #3 needs filesystem involvement


Sure it will depend on how we define the feature.
However, I am not clear about how useful are #1 & #2
as a general feature with respect to filesystems, but I
assume we could find some use cases for them.

Regarding #3 as a general feature, do we want to
consider this and the complexity associated with the
implementation?


>
>> In case of hugetlbfs it is much straight forward. Since this
>> filesystem is not like a normal filesystems and and the file sizes
>> are multiple of huge pages. The hole will be a multiple of the huge
>> page size. For this reason then should the advise be specific to
>> hugetlbfs?
> Let me paraphrase: it's simpler to implement it if it's specific to
> hugetlbfs, thus we should implement it only for hugetlbfs, and keep it
> specific to hugetlbfs.
>
> The bigger question is: do we want to continue adding to the complexity
> of hugetlbfs and increase its divergence from the core mm?

Ok,

The purpose of hugetlbfs is to enable applications to be
able to use memory in huge page sizes. Expecting that there
will be no change to its purpose other then this. The filesystem
API fallocate(), with the recent addition for hole punching support
to free pages, allows  explicit control on page
allocation  / deallocation which is useful.

It seems that the 'noautofill' feature is what is missing, with
regards to applications having explicit control on memory page
allocations using hugetlbfs. Even though the description for this
feature is not to fill holes in files, given it is filesystem semantic, but
actually the intent is to indicate not allocating pages implicitly as
the application is primarily dealing with memory allocation and
deallocation here. Is this a good enough justification?

[toc] | [prev] | [next] | [standalone]

#1637987

On Mon, May 08, 2017 at 03:12:42PM -0700, prakash.sangappa wrote:
> Regarding #3 as a general feature, do we want to
> consider this and the complexity associated with the
> implementation?

We have to.  Given that no one has exclusive access to hugetlbfs
a mount option is fundamentally the wrong interface.

[toc] | [prev] | [next] | [standalone]

#1638403

On 5/9/17 1:58 AM, Christoph Hellwig wrote:
> On Mon, May 08, 2017 at 03:12:42PM -0700, prakash.sangappa wrote:
>> Regarding #3 as a general feature, do we want to
>> consider this and the complexity associated with the
>> implementation?
> We have to.  Given that no one has exclusive access to hugetlbfs
> a mount option is fundamentally the wrong interface.


A hugetlbfs filesystem may need to be mounted for exclusive use by
an application. Note, recently the 'min_size' mount option was added
to hugetlbfs, which would reserve minimum number of huge pages
for that filesystem for use by an application. If the filesystem with
min size specified, is not setup for exclusive use by an application,
then the purpose of reserving huge pages is defeated.  The
min_size option was for use by applications like the database.

Also, I am investigating enabling hugetlbfs mounts within user
namespace's mount namespace. That would allow an application
to mount a hugetlbfs filesystem inside a namespace exclusively for
its use, running as a non root user. For this it seems like the 'min_size'
should be subject to some user limits. Anyways, mounting inside
user namespaces is  a different discussion.

So, if a filesystem has to be setup for exclusive use by an application,
then different mount options can be used for that filesystem.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web