Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1622865 > unrolled thread

[RFC] Re: Costless huge virtual memory? /dev/same, /dev/null?

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [RFC] Re: Costless huge virtual memory? /dev/same, /dev/null? Samuel Thibault <samuel.thibault@ens-lyon.org> - 2017-04-13 11:50 +0200
    Re: [RFC] Re: Costless huge virtual memory? /dev/same, /dev/null? Samuel Thibault <samuel.thibault@ens-lyon.org> - 2017-04-13 18:40 +0200
      Re: [RFC] Re: Costless huge virtual memory? /dev/same, /dev/null? Matthew Wilcox <willy@infradead.org> - 2017-04-13 18:40 +0200
        Re: [RFC] Re: Costless huge virtual memory? /dev/same, /dev/null? Samuel Thibault <samuel.thibault@ens-lyon.org> - 2017-04-13 19:10 +0200
    Re: [RFC] Re: Costless huge virtual memory? /dev/same, /dev/null? Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-04-13 20:40 +0200
      Re: [RFC] Re: Costless huge virtual memory? /dev/same, /dev/null? Samuel Thibault <samuel.thibault@ens-lyon.org> - 2017-04-13 20:50 +0200

#1622865 — [RFC] Re: Costless huge virtual memory? /dev/same, /dev/null?

[Multipart message — attachments visible in raw view] — view raw

Hello,

More than one year passed without any activity :)

I have attached a proposed patch for discussion.

Samuel

Samuel Thibault, on lun. 29 févr. 2016 17:28:35 +0100, wrote:
> I'm wondering whether we could introduce a /dev/same device to allow
> costless huge virtual memory.
> 
> The use case is the simulation of the execution of a big irregular HPC
> application, to provision memory usage, cpu time, etc. We know how much
> time each computation loop takes, and it's easy to replace them with a
> mere accounting. We'd however like to avoid having to revamp the rest
> of the code, which does allocation/memcpys/etc., by just replacing
> the allocation calls with virtual allocations, i.e. allocations which
> return addresses of buffers that one can read/write, but the values you
> read are not necessarily what you wrote, i.e. the data is not actually
> properly stored (since we don't do the actual computations that's not a
> problem).
> 
> The way we currently do this is by some folding: we map the same normal
> file several times contiguously to form the virtual allocation. By using
> a small 1MiB file, this limits memory consumption to 1MiB plus the page
> table (and fits the dumb data in a typical cache). This however creates
> one VMA per file mapping, we get limited by the 65535 VMA limit, and
> VMA lookup becomes slow.
> 
> The way I could see is to have a /dev/same device: when you open it, it
> allocates one page. When you mmap it, it maps the same page over the
> whole resulting single VMA.
> 
> This is a quite specific use case, but it seems to be easy to implement,
> and it seems to me that it could be integrated mainline. Actually I was
> thinking that /dev/null itself could be providing that service?
> (currently it returns ENODEV)
> 
> What do people think?  Is there perhaps another solution to achieve this
> that I didn't think about?
> 
> Samuel

[toc] | [next] | [standalone]

#1623171

(Ideally we'd be able to take the MAP_HUGETLB mmap flag into account to
map a single huge page repeatedly, even lowering the populating cost,
but AIUI of the current hugepage support it would be far from easy)

Samuel

[toc] | [prev] | [next] | [standalone]

#1623172

On Thu, Apr 13, 2017 at 06:29:46PM +0200, Samuel Thibault wrote:
> (Ideally we'd be able to take the MAP_HUGETLB mmap flag into account to
> map a single huge page repeatedly, even lowering the populating cost,
> but AIUI of the current hugepage support it would be far from easy)

You could implement ->map_pages instead of (or as well as) ->fault, which
would lower your costs if that is a concern.  I think the eventual intent
is that map_pages would be able to populate a PMD or even a PUD.

[toc] | [prev] | [next] | [standalone]

#1623203

Matthew Wilcox, on jeu. 13 avril 2017 09:34:11 -0700, wrote:
> On Thu, Apr 13, 2017 at 06:29:46PM +0200, Samuel Thibault wrote:
> > (Ideally we'd be able to take the MAP_HUGETLB mmap flag into account to
> > map a single huge page repeatedly, even lowering the populating cost,
> > but AIUI of the current hugepage support it would be far from easy)
> 
> You could implement ->map_pages instead of (or as well as) ->fault, which
> would lower your costs if that is a concern.

Yes it is a concern.  I was a bit afraid that implementing map_pages
would be frowned upon, and getting in first a simple yet already useful
/dev/garbage would work better :)

I'm fine with doing either.

> I think the eventual intent is that map_pages would be able to
> populate a PMD or even a PUD.

Yes.  Allocating such a big page can be a problem, though.

Samuel

[toc] | [prev] | [next] | [standalone]

#1623267

On Thu, Apr 13, 2017 at 11:42:00AM +0200, Samuel Thibault wrote:
> Hello,
> 
> More than one year passed without any activity :)
> 
> I have attached a proposed patch for discussion.

As a rule, I don't apply RFC patches, as obviously the submitter doesn't
think it is worthy of being applied :)

[toc] | [prev] | [next] | [standalone]

#1623274

Greg KH, on jeu. 13 avril 2017 20:37:56 +0200, wrote:
> On Thu, Apr 13, 2017 at 11:42:00AM +0200, Samuel Thibault wrote:
> > More than one year passed without any activity :)
> > 
> > I have attached a proposed patch for discussion.
> 
> As a rule, I don't apply RFC patches, as obviously the submitter doesn't
> think it is worthy of being applied :)

I was indeed not asking for applying it yet, I was guessing it would
raise some discussion, and code often triggers discussion :) But if
there is no objection, I'll indeed propose the patch for inclusion.

Samuel

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web