Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.soft-sys.math.mathematica > #9540
| From | David Bailey <dave@removedbailey.co.uk> |
|---|---|
| Newsgroups | comp.soft-sys.math.mathematica |
| Subject | Re: Import HTTP data in asynchronous/parallel way |
| Date | 2012-04-10 06:31 +0000 |
| Organization | Steven M. Christensen and Associates, Inc and MathTensor, Inc. |
| Message-ID | <jm0k3v$aga$1@smc.vnet.net> (permalink) |
| References | <jlp34d$14s$1@smc.vnet.net> |
On 07/04/2012 10:58, Rodrigo Murta wrote: > Hi All > > I'm testing some web scraping using Mathematica and would like to > know how to work with Import in an asynchronous/parallel way. > Now I'm using Parallelize to do that. It works but it doesn't look > like the best way to do that, due to the kernel number limitation. > > My code is like that: > result = Parallelize[Import/@urlList] > > How can I do it in another asynchronous way? Something like > backgroud process using& in bash? > I can do it inside Mathematica? I know that I could speed up a lot > my scrap with that. > > tks in advance > Murta > Since nobody else has offered a better suggestion, I'll suggest that you pull the data over using Java - which you can call via J/Link in a totally seamless way. I'd write a simple Java class that has a public, static method that takes a list of URL's to try, and uses separate threads to run them in parallel. If it kept a tally of its progress in an array: public static boolean finished[10] Then you could monitor the progress from Mathematica and read the data in the order that it got delivered. To be clear, this would involve writing and compiling a Java class, and using AddToClassPath to make it available to Mathematica's J/Link. Constructing such a thing in pure J/Link code would be very tough! David Bailey http://www.dbaileyconsultancy.co.uk
Back to comp.soft-sys.math.mathematica | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Import HTTP data in asynchronous/parallel way Rodrigo Murta <murta@rodrigomurta.com> - 2012-04-07 09:58 +0000
Re: Import HTTP data in asynchronous/parallel way David Bailey <dave@removedbailey.co.uk> - 2012-04-10 06:31 +0000
Re: Import HTTP data in asynchronous/parallel way Murta <rodrigomurtax@gmail.com> - 2012-04-11 22:19 +0000
Re: Import HTTP data in asynchronous/parallel way David Bailey <dave@removedbailey.co.uk> - 2012-04-13 09:01 +0000
csiph-web