Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Thu, 11 Aug 2011 13:52:55 +0200
From: przemolicc@poczta.fm
To: python-list@python.org
Subject: Re: String concatenation - which is the fastest way ?
References: <20110810111754.GD5045@host.pgf.com.pl> <CAPTjJmrF0GcVs0onfoCHAbMe38b5iLXgFX1R7G_RXcKxjPH5wQ@mail.gmail.com> <20110810133146.GE5045@host.pgf.com.pl> <j1ub3q$65c$1@dough.gmane.org> <20110811064030.GB4990@host.pgf.com.pl> <CAPTjJmqiiKcT7Zg-XkbrQ4r-keU9+1Kj=+sEwa4xp==2EG-5aQ@mail.gmail.com>
MIME-Version: 1.0
In-Reply-To: <CAPTjJmqiiKcT7Zg-XkbrQ4r-keU9+1Kj=+sEwa4xp==2EG-5aQ@mail.gmail.com>
User-Agent: Mutt/1.5.18 (2008-05-17)
Content-Type: text/plain; charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Precedence: list
Reply-To: przemolicc@poczta.fm
Newsgroups: comp.lang.python
Message-ID: <mailman.2172.1313063584.1164.python-list@python.org>
Lines: 93
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:11206

On Thu, Aug 11, 2011 at 11:59:31AM +0100, Chris Angelico wrote:
> On Thu, Aug 11, 2011 at 7:40 AM,  <przemolicc@poczta.fm> wrote:
> > I am not a database developer so I don't want to change the whole proce=
ss
> > of data flow between applications in my company. Another process is
> > reading this XML from particular Oracle table so I have to put the fina=
l XML there.
>=20
> I think you may be looking at a submission to
> http://www.thedailywtf.com/ soon. You seem to be working in a rather
> weird dataflow. :( Under the circumstances, you're probably going to
> want to go with the original ''.join() option.
>=20
> > This server has 256 GB of RAM so memory is not a problem.
> > Also the select which fetches the data is sorted. That is why I have to
> > carefully divide into subtasks and then merge it in correct order.
>=20
> There's no guarantee that all of that 256GB is available to you, of cours=
e.

I am the admin of this server - the memory is available for us :-)

> What may be the easiest way is to do the select in a single process,
> then partition it and use the Python multiprocessing module to split
> the job into several parts. Then you need only concatenate the handful
> of strings.

This is the way I am going to use.

> You'll need to do some serious profiling, though, to ascertain where
> the bottleneck really is. Is it actually slow doing the concatenation,
> or is it taking more time reading/writing the disk? Is it actually all
> just taking time due to RAM usage? Proper string concatenation doesn't
> need a huge amount of CPU.

I did my homework :-) - the CPU working on concatenation is a bottleneck.


Regards
Przemyslaw Bak (przemol)


















































----------------------------------------------------------------
Dziesiatki tysiecy ofert domow i mieszkan!
Ogladaj >> http://linkint.pl/f2a0c