Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.embedded > #539 > unrolled thread

kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?

Back to article view | Back to comp.os.linux.embedded

Contents

  kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives? jackbenimble <jackbenimble@mindyourhusiness.com> - 2013-10-20 05:41 +0000
    Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives? Tim Wescott <tim@seemywebsite.please> - 2013-10-20 10:43 -0500
      Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives? jackbenimble <jackbenimble@mindyourhusiness.com> - 2013-10-20 22:14 +0000
    Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives? Don Y <this@isnotme.com> - 2013-10-20 10:10 -0700
      Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives? jackbenimble <jackbenimble@mindyourhusiness.com> - 2013-10-20 22:08 +0000
        Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives? George Neuner <gneuner2@comcast.net> - 2013-10-21 03:33 -0400
      Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives? upsidedown@downunder.com - 2013-10-21 09:04 +0300
    Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives? Les Cargill <lcargill99@comcast.com> - 2013-10-20 12:48 -0500
      Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives? jackbenimble <jackbenimble@mindyourhusiness.com> - 2013-10-20 21:01 +0000
        Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives? Les Cargill <lcargill99@comcast.com> - 2013-10-20 18:19 -0500
      Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives? Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2013-10-20 17:16 -0400

#539 — kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?

I have got a high speed multi-channel data acquisition PCIE device that 
would benefit from allocating huge buffers for the dma controller. I have 
got a system with 48Gb of ram and I would like to lock down something 
like 20Gb dedicated to the dma controllers on this high-speed adc device.

running 64 bit kernel 3.8.0
	kmalloc is limited to 4Mb 
	bigphysarea patch hangs my system if I allocate anything above 1Gb
	could use mem=28G to lop off upper memory but it could get messy
	accessing that memory in user space as well as kernel

What would you suggest?

[toc] | [next] | [standalone]

#540

On Sun, 20 Oct 2013 05:41:32 +0000, jackbenimble wrote:

> I have got a high speed multi-channel data acquisition PCIE device that
> would benefit from allocating huge buffers for the dma controller. I
> have got a system with 48Gb of ram and I would like to lock down
> something like 20Gb dedicated to the dma controllers on this high-speed
> adc device.
> 
> running 64 bit kernel 3.8.0
> 	kmalloc is limited to 4Mb bigphysarea patch hangs my system if I
> 	allocate anything above 1Gb could use mem=28G to lop off upper 
memory
> 	but it could get messy accessing that memory in user space as 
well as
> 	kernel
> 
> What would you suggest?

Presumably you're running Linux.  I'm spinning off into a dark and 
mysterious land for me (meaning: I have no clue what I'm saying).

In a smaller embedded system, where you have to tell the OS (or what 
passes for one) where the memory is, the solution to this would be to 
simply lie to the OS, and then let your drivers play as they will.

Is there a way that you can configure/patch/whatever the kernel to ignore 
a big chunk of memory, and then write your drivers to talk to that 
chunk?  This would probably be easiest if you always had the same amount 
of memory available, which would leave you hardware-bound -- but it 
sounds like you're doing something special enough that you may be so 
anyway.

-- 
Tim Wescott
Control system and signal processing consulting
www.wescottdesign.com

[toc] | [prev] | [next] | [standalone]

#546

> Presumably you're running Linux.

Yup
> In a smaller embedded system, where you have to tell the OS (or what
> passes for one) where the memory is, the solution to this would be to
> simply lie to the OS, and then let your drivers play as they will.

Cant do that under Linux and still expect other apps in the system to
just play well in the sandbox. Memory allocation/use isnt the sort of
thing I want to play with - dma controllers have a way of causing problems
if left to run amok with memory the OS has other ideas about.
 
> Is there a way that you can configure/patch/whatever the kernel to
> ignore a big chunk of memory, and then write your drivers to talk to
> that chunk?

Yeah bottom and top memory allocation. But I have to make sure the memory
can be accessed in both kernel and userspace (build pages with a virtual
address for the physical block the dma has control of)

[toc] | [prev] | [next] | [standalone]

#541

On 10/19/2013 10:41 PM, jackbenimble wrote:
> I have got a high speed multi-channel data acquisition PCIE device that
> would benefit from allocating huge buffers for the dma controller. I have
> got a system with 48Gb of ram and I would like to lock down something
> like 20Gb dedicated to the dma controllers on this high-speed adc device.
>
> running 64 bit kernel 3.8.0
> 	kmalloc is limited to 4Mb
> 	bigphysarea patch hangs my system if I allocate anything above 1Gb
> 	could use mem=28G to lop off upper memory but it could get messy
> 	accessing that memory in user space as well as kernel
>
> What would you suggest?

Can you allocate (early in boot) several < 1GB buffers *expecting*
them to be contiguously located (you could examine the sources
to *know* how consecutive allocation requests will be serviced).
Then, treat them as a single contiguous object?  Once they're
*yours*, use them as *you* see fit!

(eventually releasing them in the same piecemeal fashion -- if need
be)

[Any limitations on what the DMAC can access?]

Of course, you could also dig through the sources to see *why*
the limit exists on kmalloc -- possibly implementing a similar
function with a different entry point -- or the problem "patch".
Thats the whole idea behind *having* the sources (not just,  "Hey,
I didn't have to pay any license fees for this stuff!  Yippee!")

[toc] | [prev] | [next] | [standalone]

#545

> Can you allocate (early in boot) several < 1GB buffers *expecting* them
> to be contiguously located (you could examine the sources to *know* how
> consecutive allocation requests will be serviced). Then, treat them as a
> single contiguous object?  Once they're *yours*, use them as *you* see
> fit!

with bootmem_alloc? perhaps. will need to study this more.


> (eventually releasing them in the same piecemeal fashion -- if need be)
> [Any limitations on what the DMAC can access?]

No limitation. its got 512 descriptors each house -64 bit addresses and 
byte transfer counts

> Of course, you could also dig through the sources to see *why* the limit
> exists on kmalloc -- possibly implementing a similar function with a
> different entry point -- or the problem "patch". Thats the whole idea
> behind *having* the sources (not just,  "Hey, I didn't have to pay any
> license fees for this stuff!  Yippee!")

You are right of course. Completely. I was hoping for a reason NOT to have
to go down this road. I set it aside as a LAST alternative. When all other
options have been exhausted ... seems I am nearing that point ... 

[toc] | [prev] | [next] | [standalone]

#549

On Sun, 20 Oct 2013 22:08:42 GMT, jackbenimble
<jackbenimble@mindyourhusiness.com> wrote:

>DonY wrote:
>
>> Of course, you could also dig through the sources to see *why* the limit
>> exists on kmalloc -- possibly implementing a similar function with a
>> different entry point -- or the problem "patch". Thats the whole idea
>> behind *having* the sources (not just,  "Hey, I didn't have to pay any
>> license fees for this stuff!  Yippee!")
>
>You are right of course. Completely. I was hoping for a reason NOT to have
>to go down this road. I set it aside as a LAST alternative. When all other
>options have been exhausted ... seems I am nearing that point ... 

As I understand it, bigphysarea uses alloc_bootmem_low_pages to get
the physical memory.  AFAIK, kmalloc is used only to grab little bits
of kmem for administration, so the size limit on kmalloc shouldn't
matter.

Are you sure the call to bigphysarea_setup succeeds?
Are you sure the call to bigphysarea_alloc(_pages) succeeds? 

Maybe you said this somewhere, but does the hang occur when trying to
allocate the buffer, or later trying to use the buffer?

I've only ever toyed with bigphysarea, but I *think* I recall that it
didn't actually set permissions on the buffer pages - you still had to
map the buffer into your address space just as if you'd grabbed a
bootmem block directly.

George

[toc] | [prev] | [next] | [standalone]

#548

On Sun, 20 Oct 2013 10:10:25 -0700, Don Y <this@isnotme.com> wrote:


>[Any limitations on what the DMAC can access?]

DMA controllers typically work with physical addresses, while the
program access uses virtual addresses mapped to physical memory
addresses by the MMU.

So you need some mechanism to allocate  some physical memory, for the
huge buffer, so that the OS can't directly access it (forging the top
physical address or possibly some RAMdisk system).

To access the data transferred to that memory by the DMA, you need to
map some MMU registers into that area (or at least to a part of it at
a time), so that the user/kernel mode code can access a section of
that huge buffer.

If the RAMdisk way works, then using the normal memory mapping files
should suffice.

[toc] | [prev] | [next] | [standalone]

#542

jackbenimble wrote:
> I have got a high speed multi-channel data acquisition PCIE device that
> would benefit from allocating huge buffers for the dma controller. I have
> got a system with 48Gb of ram and I would like to lock down something
> like 20Gb dedicated to the dma controllers on this high-speed adc device.
>
> running 64 bit kernel 3.8.0
> 	kmalloc is limited to 4Mb
> 	bigphysarea patch hangs my system if I allocate anything above 1Gb
> 	could use mem=28G to lop off upper memory but it could get messy
> 	accessing that memory in user space as well as kernel
>
> What would you suggest?
>

Hm. One possible approach ( I have not tried it; this is where I'd
start):

Use custom linker scripts to declare a peer memory section to .bss,
.data and the like. You'll have to declare the buffer in 'C' with
overrides for the section name to match what your linker script uses.

https://answers.launchpad.net/gcc-arm-embedded/+question/205467

seems to be a start to that. no idea what the constraints are;
you may be back to 2GB sections or something.

--
Les Cargill

[toc] | [prev] | [next] | [standalone]

#543

> Use custom linker scripts to declare a peer memory section to .bss,
> .data and the like. You'll have to declare the buffer in 'C' with
> overrides for the section name to match what your linker script uses.

Unfortunately this buffer would work for USER land programs but not for
a dma controller. memory buffers/addresses passed to a dma controller must
have certain properties:

physically contiguous and page locked (non swappable) being the most 
important.

Would your suggestion above satisfy that? In my experience memory 
allocated by the os for processes are virtually contiguous and rarely
physically so (malloc vs kmalloc).

[toc] | [prev] | [next] | [standalone]

#547

jackbenimble wrote:
>> Use custom linker scripts to declare a peer memory section to .bss,
>> .data and the like. You'll have to declare the buffer in 'C' with
>> overrides for the section name to match what your linker script uses.
>
> Unfortunately this buffer would work for USER land programs but not for
> a dma controller.

I presume you're in a kernel module, so in that case, yes it would work
exactly as you'd expect - if it works at all.

> memory buffers/addresses passed to a dma controller must
> have certain properties:
>
> physically contiguous and page locked (non swappable) being the most
> important.
>
> Would your suggestion above satisfy that?


I would think it could be made to do so, but I haven't gone through the
exersize. I know it'd be contiguous, and since it would be allocated 
only to this device driver ( through a mechanism to be named later?
I am unsure if insmod takes care of that for you, but I woould think it 
would - it's the moral equivalent of "ld" ...  ).

SFAIK ), locking just accepts an address range:
int mlock(const void *addr, size_t len); I beleive you must be 
privileged to call mlock(), and I'm not sure what addr means in kernel 
space.

I am also unsure how to interface linker scripts and such through the 
device driver build process, which may or may not use the usual
tools in the same way.

> In my experience memory
> allocated by the os for processes are virtually contiguous and rarely
> physically so (malloc vs kmalloc).
>

Right. A peer section to .bss would be all one big hunk.

--
Les Cargill

[toc] | [prev] | [next] | [standalone]

#544

On Sunday 20 October 2013 13:48, in comp.os.linux.embedded, "Les Cargill"
<lcargill99@comcast.com> wrote:

> jackbenimble wrote:
>> I have got a high speed multi-channel data acquisition PCIE device that
>> would benefit from allocating huge buffers for the dma controller. I have
>> got a system with 48Gb of ram and I would like to lock down something
>> like 20Gb dedicated to the dma controllers on this high-speed adc device.
>>
>> running 64 bit kernel 3.8.0
>> kmalloc is limited to 4Mb
>> bigphysarea patch hangs my system if I allocate anything above 1Gb
>> could use mem=28G to lop off upper memory but it could get messy
>> accessing that memory in user space as well as kernel
>>
>> What would you suggest?
>>
> 
> Hm. One possible approach ( I have not tried it; this is where I'd
> start):
> 
> Use custom linker scripts to declare a peer memory section to .bss,
> .data and the like. You'll have to declare the buffer in 'C' with
> overrides for the section name to match what your linker script uses.
[snip]

Another approach may be to
- use the mem= and memmap= kernel parameters to reserve a block of memory
for your PCIE device, and then
- hard code the reserved address (or possibly derive it from the kernel)
into the driver, so as to be able to initialize the dma controllers to the
proper addresses.

Since the kernel would be explicitly forbidden from using the memmap='ed
memory, it would be free for the OP's use. Since he would have manually
selected the address and size of the block, he can be assured of it's
placement and extent, and can code such in his drivers.


-- 
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.linux.embedded

csiph-web