Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #97379 > unrolled thread

python and ARM memory types

Back to article view | Back to comp.lang.python

Contents

  python and ARM memory types voxner.dev@gmail.com - 2015-10-03 08:07 -0700
    Re: python and ARM memory types dieter <dieter@handshake.de> - 2015-10-05 09:01 +0200
      Re: python and ARM memory types Jorgen Grahn <grahn+nntp@snipabacken.se> - 2015-10-06 09:11 +0000
        Re: python and ARM memory types Laura Creighton <lac@openend.se> - 2015-10-06 11:38 +0200
        Re: python and ARM memory types voxner <voxner.dev@gmail.com> - 2015-10-06 18:28 +0530

#97379 — python and ARM memory types

I am running python in the ARM architecture (arm64 to be exact). 

The CPU Arch I use has clusters (2 big cores in a cluster and 2 small cores in another cluster think : A57, A53). It's going to be run in Ubuntu 14.04

I am trying to run traffic that stresses the interconnects of the clusters. One example is dekker's algorithm : pin first process to 1 cluster and another process to another cluster and check the algo (using multiprocessing library).

Also I am also trying to run "streaming" traffic - the kind of traffic that you don't want in the cache because it is non-temporal. 

But how do I specify (streaming,write-combining,write-back) memory types in python ? Is there a library that I can use ? I am thinking of programming some fixed memory space (say 0x1000_000 - 0x2000_000) as "WC or WT or streaming" using the OS and then try to use the mmap facility in python. 

What set of libraries can I use ? Where should I start ? Fixed memories are discouraged so what kind of alternatives I can use ?

Thanks,
Vox

[toc] | [next] | [standalone]

#97399

voxner.dev@gmail.com writes:
> ...
> But how do I specify (streaming,write-combining,write-back) memory types in python ? Is there a library that I can use ? I am thinking of programming some fixed memory space (say 0x1000_000 - 0x2000_000) as "WC or WT or streaming" using the OS and then try to use the mmap facility in python. 

Python is quite a high level programming language - i.e. lots of things
are out of direct control of the programmer - among others memory management.

I suppose you will need an approach that gives you more control over
memory use - maybe, write your algorithms partially in the "C" programming
language.


You might find "cython" helpful to easily combine Python parts
and "C" parts.

"cython" is a compiler that compiles a source (which can use
a subset of Python and a subset of "C") into a "C" source file
which is then processed like a typical "C" source.

It takes care of a lot of the difficulties at the Python-C interface
and, thus, greatly facilitates combining Python and "C" parts.


> What set of libraries can I use ? Where should I start ? Fixed memories are discouraged so what kind of alternatives I can use ?

Note that modern operating systems virtualize memory. You will need a lot
of tricks to be able to use a specific range of physical memory in
"user level" processes -- but hopefully, you do not need to control
thing of the "physical memory level".

[toc] | [prev] | [next] | [standalone]

#97435

On Mon, 2015-10-05, dieter wrote:
> voxner.dev@gmail.com writes:
>> ...

>> But how do I specify (streaming,write-combining,write-back) memory
>> types in python ? Is there a library that I can use ? I am thinking of
>> programming some fixed memory space [...]

>
> Python is quite a high level programming language - i.e. lots of things
> are out of direct control of the programmer - among others memory management.
>
> I suppose you will need an approach that gives you more control over
> memory use - maybe, write your algorithms partially in the "C" programming
> language.

C or C++ would work.  He may want to check the assembly code that's
generated too.

> You might find "cython" helpful to easily combine Python parts
> and "C" parts.

In this case I doubt that involving Python will be helpful at all ...
There won't be a lot for it to do: all the interesting stuff will
happen at the level close to the hardware.  There's tweaking the
parameters for the benchmark, but that could be just command-line
options.

He may also want to instrument his programs using valgrind, time(1),
oprofile and perf(1), and I guess the Python interpreter would
introduce some noise there.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#97436

It is not clear to me if any of the tools that come with volutility are
going to be of any use, but I figure it is worth a look, at any rate.

https://github.com/volatilityfoundation/volatility

I seem to have replied directly, at any rate I mentioned this yesterday
but don't see it in the mailing list archives.

Laura

[toc] | [prev] | [next] | [standalone]

#97444

The main intent to use python was because it was easier to code the
algorithms. Also I was hoping to reuse the parallel algorithms
specifically iterative ones where we have some sort of control thread
that distributes data between the threads.

Now I am planning to have 2 different set of kernel with malloc that
returns WB and WT memory respectively. Then run the parallel python
program on top of that.

On 6 October 2015 at 15:08, Laura Creighton <lac@openend.se> wrote:
> It is not clear to me if any of the tools that come with volutility are
> going to be of any use, but I figure it is worth a look, at any rate.
>
> https://github.com/volatilityfoundation/volatility
>
> I seem to have replied directly, at any rate I mentioned this yesterday
> but don't see it in the mailing list archives.
>
> Laura
>
> --
> https://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web