Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #11206 > unrolled thread

Re: String concatenation - which is the fastest way ?

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: String concatenation - which is the fastest way ? przemolicc@poczta.fm - 2011-08-11 13:52 +0200

#11206 — Re: String concatenation - which is the fastest way ?

On Thu, Aug 11, 2011 at 11:59:31AM +0100, Chris Angelico wrote:
> On Thu, Aug 11, 2011 at 7:40 AM,  <przemolicc@poczta.fm> wrote:
> > I am not a database developer so I don't want to change the whole process
> > of data flow between applications in my company. Another process is
> > reading this XML from particular Oracle table so I have to put the final XML there.
> 
> I think you may be looking at a submission to
> http://www.thedailywtf.com/ soon. You seem to be working in a rather
> weird dataflow. :( Under the circumstances, you're probably going to
> want to go with the original ''.join() option.
> 
> > This server has 256 GB of RAM so memory is not a problem.
> > Also the select which fetches the data is sorted. That is why I have to
> > carefully divide into subtasks and then merge it in correct order.
> 
> There's no guarantee that all of that 256GB is available to you, of course.

I am the admin of this server - the memory is available for us :-)

> What may be the easiest way is to do the select in a single process,
> then partition it and use the Python multiprocessing module to split
> the job into several parts. Then you need only concatenate the handful
> of strings.

This is the way I am going to use.

> You'll need to do some serious profiling, though, to ascertain where
> the bottleneck really is. Is it actually slow doing the concatenation,
> or is it taking more time reading/writing the disk? Is it actually all
> just taking time due to RAM usage? Proper string concatenation doesn't
> need a huge amount of CPU.

I did my homework :-) - the CPU working on concatenation is a bottleneck.


Regards
Przemyslaw Bak (przemol)


















































----------------------------------------------------------------
Dziesiatki tysiecy ofert domow i mieszkan!
Ogladaj >> http://linkint.pl/f2a0c

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web