Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #11206

Re: String concatenation - which is the fastest way ?

Path csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path <przemolicc@poczta.fm>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.166
X-Spam-Level *
X-Spam-Evidence '*H*': 0.67; '*S*': 0.00; 'ram': 0.05; ':-)': 0.06; 'python': 0.08; 'am,': 0.12; 'developer': 0.12; 'profiling,': 0.16; 'wrote:': 0.16; 'admin': 0.18; 'memory': 0.21; 'process,': 0.21; "doesn't": 0.22; 'header:In-Reply-To:1': 0.22; '+0100,': 0.23; 'there.': 0.24; 'aug': 0.24; 'xml': 0.25; 'string': 0.26; 'thu,': 0.28; 'correct': 0.28; 'server': 0.29; 'oracle': 0.29; 'order.': 0.29; 'module': 0.30; 'strings.': 0.30; 'seem': 0.31; 'chris': 0.32; 'actually': 0.33; 'probably': 0.33; 'to:addr :python-list': 0.33; 'header:User-Agent:1': 0.34; 'option.': 0.34; 'subject: ?': 0.34; 'weird': 0.34; 'rather': 0.35; 'problem.': 0.36; 'doing': 0.36; 'another': 0.37; 'put': 0.37; 'several': 0.37; 'think': 0.38; 'easiest': 0.38; 'some': 0.38; 'subject:: ': 0.39; 'under': 0.39; "there's": 0.39; 'data': 0.39; 'why': 0.39; 'to:addr:python.org': 0.39; 'where': 0.40; 'more': 0.60; 'huge': 0.61; 'course.': 0.63; 'from:no real name:2**0': 0.63; 'charset:iso-8859-2': 0.66; 'due': 0.66; 'taking': 0.66; 'guarantee': 0.66; '11,': 0.68; 'flow': 0.68; 'carefully': 0.68; 'soon.': 0.71; 'header:Reply-To:1': 0.71; 'submission': 0.71; 'reply-to:no real name:2**0': 0.71; 'serious': 0.78; 'received:pl': 0.84; '256': 0.84; 'cpu.': 0.84; 'url:pl': 0.93
Date Thu, 11 Aug 2011 13:52:55 +0200
From przemolicc@poczta.fm
To python-list@python.org
Subject Re: String concatenation - which is the fastest way ?
References <20110810111754.GD5045@host.pgf.com.pl> <CAPTjJmrF0GcVs0onfoCHAbMe38b5iLXgFX1R7G_RXcKxjPH5wQ@mail.gmail.com> <20110810133146.GE5045@host.pgf.com.pl> <j1ub3q$65c$1@dough.gmane.org> <20110811064030.GB4990@host.pgf.com.pl> <CAPTjJmqiiKcT7Zg-XkbrQ4r-keU9+1Kj=+sEwa4xp==2EG-5aQ@mail.gmail.com>
MIME-Version 1.0
In-Reply-To <CAPTjJmqiiKcT7Zg-XkbrQ4r-keU9+1Kj=+sEwa4xp==2EG-5aQ@mail.gmail.com>
User-Agent Mutt/1.5.18 (2008-05-17)
X-Interia-Antivirus OK
X-EMID c3cafc98
Content-Type text/plain; charset="iso-8859-2"
Content-Transfer-Encoding quoted-printable
Content-Disposition inline
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.12
Precedence list
Reply-To przemolicc@poczta.fm
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.2172.1313063584.1164.python-list@python.org> (permalink)
Lines 93
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1313063584 news.xs4all.nl 23888 [2001:888:2000:d::a6]:48078
X-Complaints-To abuse@xs4all.nl
Xref x330-a1.tempe.blueboxinc.net comp.lang.python:11206

Show key headers only | View raw

On Thu, Aug 11, 2011 at 11:59:31AM +0100, Chris Angelico wrote:
> On Thu, Aug 11, 2011 at 7:40 AM,  <przemolicc@poczta.fm> wrote:
> > I am not a database developer so I don't want to change the whole process
> > of data flow between applications in my company. Another process is
> > reading this XML from particular Oracle table so I have to put the final XML there.
> 
> I think you may be looking at a submission to
> http://www.thedailywtf.com/ soon. You seem to be working in a rather
> weird dataflow. :( Under the circumstances, you're probably going to
> want to go with the original ''.join() option.
> 
> > This server has 256 GB of RAM so memory is not a problem.
> > Also the select which fetches the data is sorted. That is why I have to
> > carefully divide into subtasks and then merge it in correct order.
> 
> There's no guarantee that all of that 256GB is available to you, of course.

I am the admin of this server - the memory is available for us :-)

> What may be the easiest way is to do the select in a single process,
> then partition it and use the Python multiprocessing module to split
> the job into several parts. Then you need only concatenate the handful
> of strings.

This is the way I am going to use.

> You'll need to do some serious profiling, though, to ascertain where
> the bottleneck really is. Is it actually slow doing the concatenation,
> or is it taking more time reading/writing the disk? Is it actually all
> just taking time due to RAM usage? Proper string concatenation doesn't
> need a huge amount of CPU.

I did my homework :-) - the CPU working on concatenation is a bottleneck.


Regards
Przemyslaw Bak (przemol)


















































----------------------------------------------------------------
Dziesiatki tysiecy ofert domow i mieszkan!
Ogladaj >> http://linkint.pl/f2a0c

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: String concatenation - which is the fastest way ? przemolicc@poczta.fm - 2011-08-11 13:52 +0200

csiph-web