Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #24525 > unrolled thread

Slow output

Back to article view | Back to comp.lang.python

Contents

  Slow output subhabangalore@gmail.com - 2012-06-27 10:33 -0700
    Re: Slow output MRAB <python@mrabarnett.plus.com> - 2012-06-27 19:06 +0100
      Re: Slow output Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-06-27 20:27 +0000
    Re: Slow output Jean-Michel Pichavant <jeanmichel@sequans.com> - 2012-06-27 20:59 +0200
    Re: Slow output alex23 <wuwei23@gmail.com> - 2012-06-27 19:05 -0700
    Re: Slow output subhabangalore@gmail.com - 2012-06-28 07:11 -0700
    Re: Slow output Roel Schroeven <roel@roelschroeven.net> - 2012-06-28 21:14 +0200

#24525 — Slow output

Dear Group,
I am Sri Subhabrata Banerjee writing from India. I am running a small program which exploits around 12 1 to 2 KB .txt files. I am using MS Windows XP Service Pack 3 and Python 2.6 where IDLE is GUI. The text is plain ASCII text. The RAM of the machine is around 2 GB. To run the program the machine is becoming dead slow and it is executing only after several minutes. I am shocked as I have run hugely loaded Statistical Learning stuff only on 516MB RAM on Python using Windows XP only. And that too on a laptop. 
I am not getting. I discussed the issue with my system administrator and he increased the RAM to 8GB result improved by 30% but not satisfactory speed. May any one suggest me what may be the likely issue?

Thanking You in Advance,
Subhabrata Banerjee. 


 

[toc] | [next] | [standalone]

#24528

On 27/06/2012 18:33, subhabangalore@gmail.com wrote:
> Dear Group,
> I am Sri Subhabrata Banerjee writing from India. I am running a small
> program which exploits around 12 1 to 2 KB .txt files. I am using MS
> Windows XP Service Pack 3 and Python 2.6 where IDLE is GUI. The text
> is plain ASCII text. The RAM of the machine is around 2 GB. To run
> the program the machine is becoming dead slow and it is executing
> only after several minutes. I am shocked as I have run hugely loaded
> Statistical Learning stuff only on 516MB RAM on Python using Windows
> XP only. And that too on a laptop. I am not getting. I discussed the
> issue with my system administrator and he increased the RAM to 8GB
> result improved by 30% but not satisfactory speed. May any one
> suggest me what may be the likely issue?
>
Could you post the program here (you say that it's small) so that we
can see what you're trying to do.

[toc] | [prev] | [next] | [standalone]

#24537

On Wed, 27 Jun 2012 19:06:08 +0100, MRAB wrote:

> On 27/06/2012 18:33, subhabangalore@gmail.com wrote:
>> Dear Group,
>> I am Sri Subhabrata Banerjee writing from India. I am running a small
>> program which exploits around 12 1 to 2 KB .txt files. I am using MS
>> Windows XP Service Pack 3 and Python 2.6 where IDLE is GUI. The text is
>> plain ASCII text. The RAM of the machine is around 2 GB. To run the
>> program the machine is becoming dead slow and it is executing only
>> after several minutes.
[...]
> Could you post the program here (you say that it's small) so that we can
> see what you're trying to do.

I bet that he is building up long strings using repeated string 
concatenation:

s = ''
for item in many_items:
    s += item


instead of accumulating them into a list, then joining them all at once.

Repeated string concatenation can be *painfully* slow, especially under 
Windows. (For some reason, the details of Windows memory management 
sometimes prevents the string concat optimization from working.)


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#24532

MRAB wrote:
> On 27/06/2012 18:33, subhabangalore@gmail.com wrote:
>> Dear Group,
>> I am Sri Subhabrata Banerjee writing from India. I am running a small
>> program which exploits around 12 1 to 2 KB .txt files. I am using MS
>> Windows XP Service Pack 3 and Python 2.6 where IDLE is GUI. The text
>> is plain ASCII text. The RAM of the machine is around 2 GB. To run
>> the program the machine is becoming dead slow and it is executing
>> only after several minutes. I am shocked as I have run hugely loaded
>> Statistical Learning stuff only on 516MB RAM on Python using Windows
>> XP only. And that too on a laptop. I am not getting. I discussed the
>> issue with my system administrator and he increased the RAM to 8GB
>> result improved by 30% but not satisfactory speed. May any one
>> suggest me what may be the likely issue?
>>
> Could you post the program here (you say that it's small) so that we
> can see what you're trying to do.
Additionaly, you may want to look at 
http://docs.python.org/library/profile.html
You may spot quite quickly where your program is dying. It's worth the 
try, it's doesn't take long to setup the profile.

JM

[toc] | [prev] | [next] | [standalone]

#24572

On Jun 28, 3:33 am, subhabangal...@gmail.com wrote:
> May any one suggest me what may be the likely issue?

In situations like this, it always helps to see your code, especially
if you can reduce it down to only the part doing the loading.

One thing that can help reduce memory usage is to replace lists/list
comprehensions with generators. For example, this loads the entire
file into memory:

    for line in open('big.txt').readlines():
         # do something to line

While this only loads one line at a time:

    for line in open('big.txt'):
        # do something to line

You can also do the same with multiple files by creating a generator
to return their content a line at a time:

    filenames = ['a.txt', 'b.txt', 'c.txt']
    files = (open(name) for name in filenames)
    lines = (line for file in files for line in file)

    for line in lines:
        # do something to line

I highly recommend David Beazley's "Generator Tricks for Systems
Programmers" for more techniques like this.

[toc] | [prev] | [next] | [standalone]

#24613

On Wednesday, June 27, 2012 11:03:44 PM UTC+5:30, (unknown) wrote:
> Dear Group,
> I am Sri Subhabrata Banerjee writing from India. I am running a small program which exploits around 12 1 to 2 KB .txt files. I am using MS Windows XP Service Pack 3 and Python 2.6 where IDLE is GUI. The text is plain ASCII text. The RAM of the machine is around 2 GB. To run the program the machine is becoming dead slow and it is executing only after several minutes. I am shocked as I have run hugely loaded Statistical Learning stuff only on 516MB RAM on Python using Windows XP only. And that too on a laptop. 
> I am not getting. I discussed the issue with my system administrator and he increased the RAM to 8GB result improved by 30% but not satisfactory speed. May any one suggest me what may be the likely issue?
> 
> Thanking You in Advance,
> Subhabrata Banerjee.

Dear Group,
Thank you for your kind reply and teaching me some nice coding options. I would port your suggestions and see if it improves. Unfortuantely I can not share the code as it has slight propietary issue, but I was debugging the code on myself, I was seeing that if I write the outputs in the file then it is giving fast results, but if I print it to the console -which is a list of big size, but well even I print all the files of 1KB each I have 12 files only, why a print command would make it so slow? I have searched processed made big n-grams big calculations across even UTF-8 encoding data sets Python gave result in fragment of seconds.
Regards,
Subhabrata Banerjee. 

[toc] | [prev] | [next] | [standalone]

#24633

subhabangalore@gmail.com schreef:
> Dear Group,
> I am Sri Subhabrata Banerjee writing from India. I am running a small
> program which exploits around 12 1 to 2 KB .txt files. I am using MS
> Windows XP Service Pack 3 and Python 2.6 where IDLE is GUI. The text
> is plain ASCII text. The RAM of the machine is around 2 GB. To run
> the program the machine is becoming dead slow and it is executing
> only after several minutes. I am shocked as I have run hugely loaded
> Statistical Learning stuff only on 516MB RAM on Python using Windows
> XP only. And that too on a laptop. I am not getting. I discussed the
> issue with my system administrator and he increased the RAM to 8GB
> result improved by 30% but not satisfactory speed. May any one
> suggest me what may be the likely issue?

Does the program generate a lot of text? Do you run the program in IDLE? 
IDLE gets very slow when it has to show a lot of text. Try running the 
program from the command line to see if that offers any improvement.

Best regards,
Roel

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web