Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #9499 > unrolled thread

Import HTTP data in asynchronous/parallel way

Started byRodrigo Murta <murta@rodrigomurta.com>
First post2012-04-07 09:58 +0000
Last post2012-04-13 09:01 +0000
Articles 4 — 3 participants

Back to article view | Back to comp.soft-sys.math.mathematica


Contents

  Import HTTP data in asynchronous/parallel way Rodrigo Murta <murta@rodrigomurta.com> - 2012-04-07 09:58 +0000
    Re: Import HTTP data in asynchronous/parallel way David Bailey <dave@removedbailey.co.uk> - 2012-04-10 06:31 +0000
      Re: Import HTTP data in asynchronous/parallel way Murta <rodrigomurtax@gmail.com> - 2012-04-11 22:19 +0000
        Re: Import HTTP data in asynchronous/parallel way David Bailey <dave@removedbailey.co.uk> - 2012-04-13 09:01 +0000

#9499 — Import HTTP data in asynchronous/parallel way

FromRodrigo Murta <murta@rodrigomurta.com>
Date2012-04-07 09:58 +0000
SubjectImport HTTP data in asynchronous/parallel way
Message-ID<jlp34d$14s$1@smc.vnet.net>
Hi All

    I'm testing some web scraping using Mathematica and would like to
know how to work with Import in an asynchronous/parallel way.
    Now I'm using Parallelize to do that. It works but it doesn't look
like the best way to do that, due to the kernel number limitation.

   My code is like that:
   result = Parallelize[Import/@urlList]

  How can I do it in another asynchronous way? Something like
backgroud process using & in bash?
  I can do it inside Mathematica? I know that I could speed up a lot
my scrap with that.

tks in advance
Murta

[toc] | [next] | [standalone]


#9540

FromDavid Bailey <dave@removedbailey.co.uk>
Date2012-04-10 06:31 +0000
Message-ID<jm0k3v$aga$1@smc.vnet.net>
In reply to#9499
On 07/04/2012 10:58, Rodrigo Murta wrote:
> Hi All
>
>      I'm testing some web scraping using Mathematica and would like to
> know how to work with Import in an asynchronous/parallel way.
>      Now I'm using Parallelize to do that. It works but it doesn't look
> like the best way to do that, due to the kernel number limitation.
>
>     My code is like that:
>     result = Parallelize[Import/@urlList]
>
>    How can I do it in another asynchronous way? Something like
> backgroud process using&  in bash?
>    I can do it inside Mathematica? I know that I could speed up a lot
> my scrap with that.
>
> tks in advance
> Murta
>
Since nobody else has offered a better suggestion, I'll suggest that you 
pull the data over using Java - which you can call via J/Link in a 
totally seamless way.

I'd write a simple Java class that has a public, static method that 
takes a list of URL's to try, and uses separate threads to run them in 
parallel. If it kept a tally of its progress in an array:

public static boolean finished[10]

Then you could monitor the progress from Mathematica and read the data 
in the order that it got delivered.

To be clear, this would involve writing and compiling a Java class, and 
using AddToClassPath to make it available to Mathematica's J/Link. 
Constructing such a thing in pure J/Link code would be very tough!

David Bailey
http://www.dbaileyconsultancy.co.uk

[toc] | [prev] | [next] | [standalone]


#9552

FromMurta <rodrigomurtax@gmail.com>
Date2012-04-11 22:19 +0000
Message-ID<jm501q$2ln$1@smc.vnet.net>
In reply to#9540
On Apr 10, 3:31 am, David Bailey <d...@removedbailey.co.uk> wrote:
> On 07/04/2012 10:58, Rodrigo Murta wrote:
>
>
>
>
>
>
>
> > Hi All
>
> >      I'm testing some web scraping using Mathematica and would like to
> > know how to work with Import in an asynchronous/parallel way.
> >      Now I'm using Parallelize to do that. It works but it doesn't look
> > like the best way to do that, due to the kernel number limitation.
>
> >     My code is like that:
> >     result = Parallelize[Import/@urlList]
>
> >    How can I do it in another asynchronous way? Something like
> > backgroud process using&  in bash?
> >    I can do it inside Mathematica? I know that I could speed up a lot
> > my scrap with that.
>
> > tks in advance
> > Murta
>
> Since nobody else has offered a better suggestion, I'll suggest that you
> pull the data over using Java - which you can call via J/Link in a
> totally seamless way.
>
> I'd write a simple Java class that has a public, static method that
> takes a list of URL's to try, and uses separate threads to run them in
> parallel. If it kept a tally of its progress in an array:
>
> public static boolean finished[10]
>
> Then you could monitor the progress from Mathematica and read the data
> in the order that it got delivered.
>
> To be clear, this would involve writing and compiling a Java class, and
> using AddToClassPath to make it available to Mathematica's J/Link.
> Constructing such a thing in pure J/Link code would be very tough!
>
> David Baileyhttp://www.dbaileyconsultancy.co.uk

Hi David,
First of all tks for your answer.
For me the problem is that I know zero of Java, I hopped to be capable
to solve it inside Mathematica.
I get a little bit frustrated in see that Mathematica is Multi Kerner,
but not Mult Thread with one Kernel, in the sence that inside one
Kernel it's not possible to have more than one process at the same
time. This is essencial to handle internet process.
Someone could correct me if I'm wrong!
tks again
Murta

[toc] | [prev] | [next] | [standalone]


#9611

FromDavid Bailey <dave@removedbailey.co.uk>
Date2012-04-13 09:01 +0000
Message-ID<jm8q14$j54$1@smc.vnet.net>
In reply to#9552
On 11/04/2012 23:19, Murta wrote:
> Hi David,
> First of all tks for your answer.
> For me the problem is that I know zero of Java, I hopped to be capable
> to solve it inside Mathematica.
> I get a little bit frustrated in see that Mathematica is Multi Kerner,
> but not Mult Thread with one Kernel, in the sence that inside one
> Kernel it's not possible to have more than one process at the same
> time. This is essencial to handle internet process.
> Someone could correct me if I'm wrong!
> tks again
> Murta
>

You are almost correct, but each kernel has two threads, one of which is 
used for interactions with the FrontEnd, Java, etc - basically to handle 
asynchronous callbacks.

Don't forget that you might be able to find a command line utility that 
would do what you want, and you could then invoke it from Mathematica in 
a pretty seamless way.

If you are working on problems of this sort, it may be worth learning a 
little Java.

David Bailey
http://www.dbaileyconsultancy.co.uk

[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web