Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.ruby > #3557 > unrolled thread

Fast way to move C array to ruby (rb_ary_new4)?

Back to article view | Back to comp.lang.ruby

Contents

  Fast way to move C array to ruby (rb_ary_new4)? Ville Sipola <ville_sipola@hotmail.com> - 2011-04-27 05:46 -0500
    Re: Fast way to move C array to ruby (rb_ary_new4)? Robert Klemme <shortcutter@googlemail.com> - 2011-04-27 07:08 -0500
      Re: Fast way to move C array to ruby (rb_ary_new4)? Robert Klemme <shortcutter@googlemail.com> - 2011-04-27 08:06 -0500
        Re: Fast way to move C array to ruby (rb_ary_new4)? Cameron McBride <cameron.mcbride@gmail.com> - 2011-04-27 15:02 -0500
          Re: Fast way to move C array to ruby (rb_ary_new4)? Ville Sipola <ville_sipola@hotmail.com> - 2011-04-27 17:01 -0500
            Re: Fast way to move C array to ruby (rb_ary_new4)? Ville Sipola <ville_sipola@hotmail.com> - 2011-05-06 14:47 -0500
    Re: Fast way to move C array to ruby (rb_ary_new4)? Brian Candler <b.candler@pobox.com> - 2011-04-27 11:15 -0500

#3557 — Fast way to move C array to ruby (rb_ary_new4)?

Good day

I have a number of large arrays in a C module, and I'd need to move them
to ruby as quickly as possible.

Currently I use the following type of an approach:

instance_array_in_ruby = rb_iv_get(calling_class, "@x");
double c_array[1000000];

for(i=0;i<1000000;i++)
{
   rb_ary_store(instance_array_in_ruby, i, rb_float_new(c_array[i]));
}

But that's slow, apparently due to the large number of rb_float_new:s.

There is a function called rb_ary_new4 that would seem to fit
the purpose of quickly moving an entire C array to ruby. Only I'm unable
to get it work, as it, too requires VALUE:s.
I've tried many things including mumbo-jumbo like
rb_iv_set(calling_class,"@x",rb_ary_new4(1000000, (VALUE)c_array));, but
I'm only managing to get segfaults or solutions that still need a
million rb_something_new:s.

Would you know an efficient way of moving a large data from C array to
ruby array?

-- 
Posted via http://www.ruby-forum.com/.

[toc] | [next] | [standalone]

#3559

On Wed, Apr 27, 2011 at 12:46 PM, Ville Sipola <ville_sipola@hotmail.com> wrote:
> Good day
>
> I have a number of large arrays in a C module, and I'd need to move them
> to ruby as quickly as possible.
>
> Currently I use the following type of an approach:
>
> instance_array_in_ruby = rb_iv_get(calling_class, "@x");
> double c_array[1000000];
>
> for(i=0;i<1000000;i++)
> {
>   rb_ary_store(instance_array_in_ruby, i, rb_float_new(c_array[i]));
> }
>
> But that's slow, apparently due to the large number of rb_float_new:s.
>
> There is a function called rb_ary_new4 that would seem to fit
> the purpose of quickly moving an entire C array to ruby. Only I'm unable
> to get it work, as it, too requires VALUE:s.
> I've tried many things including mumbo-jumbo like
> rb_iv_set(calling_class,"@x",rb_ary_new4(1000000, (VALUE)c_array));, but
> I'm only managing to get segfaults or solutions that still need a
> million rb_something_new:s.
>
> Would you know an efficient way of moving a large data from C array to
> ruby array?

What about leaving it in C, i.e. wrapping in a specific type which
presents an Array (or Enumerable) like API and uses your C array
internally?  A similar approach is taken by ENV (you can find the code
in hash.c).

Kind regards

robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[toc] | [prev] | [next] | [standalone]

#3560

On Wed, Apr 27, 2011 at 2:08 PM, Robert Klemme
<shortcutter@googlemail.com> wrote:
> On Wed, Apr 27, 2011 at 12:46 PM, Ville Sipola <ville_sipola@hotmail.com> wrote:

>> Would you know an efficient way of moving a large data from C array to
>> ruby array?
>
> What about leaving it in C, i.e. wrapping in a specific type which
> presents an Array (or Enumerable) like API and uses your C array
> internally?  A similar approach is taken by ENV (you can find the code
> in hash.c).

PS: Of course whether that is more or less efficient than your
approach totally depends on the usage pattern.  If only few items in
those arrays are accessed from Ruby land the wrapping approach is
likely cheaper.  If all items are accessed in the majority of cases
then probably your original approach is better.

I'd also look into NArray which might help you here.

Kind regards

robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[toc] | [prev] | [next] | [standalone]

#3573

On Wed, Apr 27, 2011 at 08:06, Robert Klemme <shortcutter@googlemail.com> wrote:
> On Wed, Apr 27, 2011 at 2:08 PM, Robert Klemme
> <shortcutter@googlemail.com> wrote:
>> On Wed, Apr 27, 2011 at 12:46 PM, Ville Sipola <ville_sipola@hotmail.com> wrote:
>>> Would you know an efficient way of moving a large data from C array to
>>> ruby array?

> I'd also look into NArray which might help you here.

I second this suggestion. If you don't mind the NArray dependence,
this is extremely easy using NArray as it keeps it in the C array
structure.

Cameron

[toc] | [prev] | [next] | [standalone]

#3585

That's a lot of answers in a short time. Thank you all!

The kind of hack that Brian Candler mentioned (writing directly to
struct RFloat) was one of the ways that I already tried to accomplish,
but if it's likely to get broken in future, then I guess it's not a
solution to fight for.

And for Robert's second answer: yes, I need the whole data back, I'm
sending it all to other functions.

I guess I'll try them all, starting with the NArray.

Now I only have to figure out how to use that. From narray.c I get the
impression that they can be read into C by creating a struct NARRAY and
then using the command GetNArray, right? But how would you write into it
in C?

Thank you for the help.

Ville

-- 
Posted via http://www.ruby-forum.com/.

[toc] | [prev] | [next] | [standalone]

#4051

Ville Sipola wrote:
> Now I only have to figure out how to use that.

Now that I have, I thought it a good idea to describe how it was done,
so that others reading the post might be able to use the information.

So this was my method.

The writing part is a bit of a hack. Masahiro Tanaka himself advices
against this approach in here: https://www.ruby-forum.com/topic/83484,
but I'm yiet to find a better solution. If there is a function for
writing c data in an narray in a safe way, I'd like to hear about it.

First, in Ruby I did something like

@x=NArray.float(@number_of_samples).fill(some_value)

to make sure that the narray is of correct size.

Then in C:
static VALUE writingfunction(VALUE class,VALUE x)
{
    struct NARRAY *xarray;
    GetNArray(x, xarray);
    double *floatpointerx = xarray->ptr;

    double c_value;

    for (counter = 0; counter < number_of_samples; counter++)
    {
       (do the actual calculation)
       floatpointerx[samplecounter] = c_value; (This is the illegal
hack)
    }
}

To read in C (this I believe is completely ok, please correct if I'm
wrong.)

static VALUE readingfunction(VALUE class, VALUE x)
{
    struct NARRAY *xarray;
    GetNArray(x, xarray);
    double *floatpointerx = xarray->ptr;
    for (counter = 0; counter < xarray->total; counter++)
    {
        a_function_that_takes_c_double(floatpointerx[counter1]);
    }
}

-- 
Posted via http://www.ruby-forum.com/.

[toc] | [prev] | [next] | [standalone]

#3563

Ville Sipola wrote in post #995293:
> instance_array_in_ruby = rb_iv_get(calling_class, "@x");
> double c_array[1000000];
>
> for(i=0;i<1000000;i++)
> {
>    rb_ary_store(instance_array_in_ruby, i, rb_float_new(c_array[i]));
> }
>
> But that's slow, apparently due to the large number of rb_float_new:s.

I'm afraid that's bound to be the case, given that Floats are not 
immediate values and therefore each one has to be allocated on the heap 
(unless your data happens to have many instances of identical floats, 
and could share the references)

Robert's wrapping sounds like the best way forward, perhaps with some 
memoizing (i.e. lazy creation of Float objects)

It might also help a little if you pre-allocate the array:

a = Array.new(1000000)

For more of a hack, try:

a = Array.new(1000000) { 0.0+0.0 }

Then maybe you could walk the array and overwrite the RFloat structures 
directly (ugh). Note that the 0.0+0.0 frig was required so that the same 
object_id doesn't get used in each element, and that's not guaranteed to 
work in future (e.g. if Ruby did more aggressive constant folding)

-- 
Posted via http://www.ruby-forum.com/.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.ruby

csiph-web