Re: String concatenation - which is the fastest way ?

From	Stefan Behnel <stefan_ml@behnel.de>
Subject	Re: String concatenation - which is the fastest way ?
Date	2011-08-12 11:25 +0200
References	(4 earlier) <20110811064030.GB4990@host.pgf.com.pl> <CAPTjJmqiiKcT7Zg-XkbrQ4r-keU9+1Kj=+sEwa4xp==2EG-5aQ@mail.gmail.com> <20110811134613.GE4990@host.pgf.com.pl> <CAPTjJmr7qDeGU+ttEXsdBOASqZ5w1mYgkg18o+RtFp55HHp8Ew@mail.gmail.com> <20110811143923.GG4990@host.pgf.com.pl>
Newsgroups	comp.lang.python
Message-ID	<mailman.2213.1313141121.1164.python-list@python.org> (permalink)

Show all headers | View raw

przemolicc@poczta.fm, 11.08.2011 16:39:
> On Thu, Aug 11, 2011 at 02:48:43PM +0100, Chris Angelico wrote:
>> On Thu, Aug 11, 2011 at 2:46 PM,<przemolicc@poczta.fm>  wrote:
>>> This is the way I am going to use.
>>> But what is the best data type to hold so many rows and then operate on them ?
>>>
>>
>> List of strings. Take it straight from your Oracle interface and work
>> with it directly.
>
> Can I use this list in the following way ?
> subprocess_1 - run on list between 1 and 10000
> subprocess_2 - run on list between 10001 and 20000
> subprocess_3 - run on list between 20001 and 30000
> etc
> ...

Sure. Just read the data as it comes in from the database and fill up a 
chunk, then hand that on to a process. You can also distribute it in 
smaller packets, just check what size gives the best throughput.

Still, I'd give each work parcel a number and then reorder the results 
while collecting them, that allows you to vary the chunk size and the 
process size independently, without having to wait for a process that 
happens to take longer.

Stefan

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: String concatenation - which is the fastest way ? Stefan Behnel <stefan_ml@behnel.de> - 2011-08-12 11:25 +0200

csiph-web