Re: String concatenation - which is the fastest way ?

Date	2011-08-11 13:52 +0200
From	przemolicc@poczta.fm
Subject	Re: String concatenation - which is the fastest way ?
References	(1 earlier) <CAPTjJmrF0GcVs0onfoCHAbMe38b5iLXgFX1R7G_RXcKxjPH5wQ@mail.gmail.com> <20110810133146.GE5045@host.pgf.com.pl> <j1ub3q$65c$1@dough.gmane.org> <20110811064030.GB4990@host.pgf.com.pl> <CAPTjJmqiiKcT7Zg-XkbrQ4r-keU9+1Kj=+sEwa4xp==2EG-5aQ@mail.gmail.com>
Newsgroups	comp.lang.python
Message-ID	<mailman.2172.1313063584.1164.python-list@python.org> (permalink)

Show all headers | View raw

On Thu, Aug 11, 2011 at 11:59:31AM +0100, Chris Angelico wrote:
> On Thu, Aug 11, 2011 at 7:40 AM,  <przemolicc@poczta.fm> wrote:
> > I am not a database developer so I don't want to change the whole process
> > of data flow between applications in my company. Another process is
> > reading this XML from particular Oracle table so I have to put the final XML there.
> 
> I think you may be looking at a submission to
> http://www.thedailywtf.com/ soon. You seem to be working in a rather
> weird dataflow. :( Under the circumstances, you're probably going to
> want to go with the original ''.join() option.
> 
> > This server has 256 GB of RAM so memory is not a problem.
> > Also the select which fetches the data is sorted. That is why I have to
> > carefully divide into subtasks and then merge it in correct order.
> 
> There's no guarantee that all of that 256GB is available to you, of course.

I am the admin of this server - the memory is available for us :-)

> What may be the easiest way is to do the select in a single process,
> then partition it and use the Python multiprocessing module to split
> the job into several parts. Then you need only concatenate the handful
> of strings.

This is the way I am going to use.

> You'll need to do some serious profiling, though, to ascertain where
> the bottleneck really is. Is it actually slow doing the concatenation,
> or is it taking more time reading/writing the disk? Is it actually all
> just taking time due to RAM usage? Proper string concatenation doesn't
> need a huge amount of CPU.

I did my homework :-) - the CPU working on concatenation is a bottleneck.


Regards
Przemyslaw Bak (przemol)


















































----------------------------------------------------------------
Dziesiatki tysiecy ofert domow i mieszkan!
Ogladaj >> http://linkint.pl/f2a0c

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: String concatenation - which is the fastest way ? przemolicc@poczta.fm - 2011-08-11 13:52 +0200

csiph-web