Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #73368 > unrolled thread

Not Responding When Dealing with Large Data

Back to article view | Back to comp.lang.python

Contents

  Not Responding When Dealing with Large Data cutey Love <cuteywithlove@gmail.com> - 2014-06-18 10:20 -0700
    Re: Not Responding When Dealing with Large Data Philip Dexter <philip.dexter@gmail.com> - 2014-06-18 13:36 -0400
    Re: Not Responding When Dealing with Large Data Tim <jtim.arnold@gmail.com> - 2014-06-18 11:37 -0700
    Re: Not Responding When Dealing with Large Data John Gordon <gordon@panix.com> - 2014-06-18 19:10 +0000
    Re: Not Responding When Dealing with Large Data Michael Torrie <torriem@gmail.com> - 2014-06-18 13:25 -0600
    Re: Not Responding When Dealing with Large Data Paul McNett <paul@mcnettware.com> - 2014-06-18 12:23 -0700
    Re: Not Responding When Dealing with Large Data cutey Love <cuteywithlove@gmail.com> - 2014-06-18 15:32 -0700
      Re: Not Responding When Dealing with Large Data Paul McNett <paul@mcnettware.com> - 2014-06-18 15:48 -0700
      Re: Not Responding When Dealing with Large Data Ian Kelly <ian.g.kelly@gmail.com> - 2014-06-18 17:06 -0600
        Re: Not Responding When Dealing with Large Data cutey Love <cuteywithlove@gmail.com> - 2014-06-19 01:17 -0700
          Re: Not Responding When Dealing with Large Data MRAB <python@mrabarnett.plus.com> - 2014-06-19 12:25 +0100
            Re: Not Responding When Dealing with Large Data Peter Pearson <ppearson@nowhere.invalid> - 2014-06-19 16:21 +0000
              Re: Not Responding When Dealing with Large Data MRAB <python@mrabarnett.plus.com> - 2014-06-19 17:54 +0100

#73368 — Not Responding When Dealing with Large Data

I'm trying to read in 100000 lines of text, use some functions to edit them and then return a new list.

The problem is my program always goes not responding when the amount of lines are a high number.

I don't care how long the program takes to work, just need it to stop crashing?

[toc] | [next] | [standalone]

#73369

On Wed, Jun 18, 2014 at 1:20 PM, cutey Love <cuteywithlove@gmail.com> wrote:
> I'm trying to read in 100000 lines of text, use some functions to edit them and then return a new list.
>
> The problem is my program always goes not responding when the amount of lines are a high number.
>
> I don't care how long the program takes to work, just need it to stop crashing?


What are you doing with the data? Try reading in the file chunks at a time

[toc] | [prev] | [next] | [standalone]

#73370

On Wednesday, June 18, 2014 1:20:13 PM UTC-4, cutey Love wrote:
> I'm trying to read in 100000 lines of text, use some functions to edit them and then return a new list.
> The problem is my program always goes not responding when the amount of lines are a high number.
> I don't care how long the program takes to work, just need it to stop crashing?

Please show some code. What kind of text (how long are the lines roughly)?
thanks,
--Tim

[toc] | [prev] | [next] | [standalone]

#73371

In <00d330e3-fc8a-4b7e-b2bd-f1a48bc335c1@googlegroups.com> cutey Love <cuteywithlove@gmail.com> writes:

> The problem is my program always goes not responding when the amount of
> lines are a high number.

> I don't care how long the program takes to work, just need it to stop
> crashing?

How long have you tried waiting when the program stops responding? Perhaps
the program just has a lot of work to do.

You might try adding some print statements throughout the program, so it
produces observable output instead of appearing to be unresponsive.

Why do you say the program 'crashes'?  Does it actually crash?

-- 
John Gordon         Imagine what it must be like for a real medical doctor to
gordon@panix.com    watch 'House', or a real serial killer to watch 'Dexter'.

[toc] | [prev] | [next] | [standalone]

#73376

On 06/18/2014 11:20 AM, cutey Love wrote:
> I'm trying to read in 100000 lines of text, use some functions to
> edit them and then return a new list.
> 
> The problem is my program always goes not responding when the amount
> of lines are a high number.
> 
> I don't care how long the program takes to work, just need it to stop
> crashing?

Welcome to the world of GUI programming. GUI programs are event-driven.
 When an event happens your code runs to handle the event.  While you
are handling an event, the rest of the GUI cannot run, until you return
control to the main loop.  So when you have to do long-running tasks,
you need to find a way to return control to the main loop while your
work continues.  There are basically two ways to do this. One is to
spawn a thread to do the work, the other is to use asynchronous
programming, depending on what you need to get done.  The problem with
threads is that you cannot directly make GUI calls. Instead you have to
set up a message queue of some sort, or maybe use flag variables, and
then have a callback that fires periodically, regularly with a some kind
of clock, or during the idle portion.

Anyway your question is light on details, so my replay is rather light.
 Hopefully you are now pointed in the right direction for asking
questions at least.  A quick google search reveals this which is pretty
much the sort of thing you wish to do:

http://stackoverflow.com/quhttp://stackoverflow.com/questions/16745507/tkinter-how-to-use-threads-to-preventing-main-event-loop-from-freezingestions/16745507/tkinter-how-to-use-threads-to-preventing-main-event-loop-from-freezing

[toc] | [prev] | [next] | [standalone]

#73377

On 6/18/14, 10:20 AM, cutey Love wrote:
> I'm trying to read in 100000 lines of text, use some functions to edit them and then return a new list.

How is it that you are opening the file, and iterating the contents?

> The problem is my program always goes not responding when the amount of lines are a high number.

What do you mean, "goes not responding"? What environment are you 
running in? Windows and IDLE?

> I don't care how long the program takes to work, just need it to stop crashing?

Note that "not responding" isn't the same as crashing. If your program 
crashed, it would likely print on your terminal something like:

Traceback (most recent call last):
... lots of lines including useful information about the problem

or

Segmentation Fault

If it just goes "not responding" for a while in your tight loop, it 
could simply mean just that: your IDE or whatever is focusing so hard on 
running your program that there aren't any cycles to say "still here" to 
the OS, so the OS has no clue what's going on and says to the user "not 
responding".

Have you waited to see if it ever completes?

Paul

[toc] | [prev] | [next] | [standalone]

#73382

Hi, thanks for the replies,

I mean windows displays "Not responding close the program now" 

How can I do it asynconistrically?

It's simple code just open file, loop through line by line and do some comparons with the string.

On Wednesday, June 18, 2014 6:20:13 PM UTC+1, cutey Love wrote:
> I'm trying to read in 100000 lines of text, use some functions to edit them and then return a new list.
> 
> 
> 
> The problem is my program always goes not responding when the amount of lines are a high number.
> 
> 
> 
> I don't care how long the program takes to work, just need it to stop crashing?

[toc] | [prev] | [next] | [standalone]

#73383

On 6/18/14, 3:32 PM, cutey Love wrote:
> Hi, thanks for the replies,
>
> I mean windows displays "Not responding close the program now"
>
> How can I do it asynconistrically?
>
> It's simple code just open file, loop through line by line and do some comparons with the string.

To get anything better from us than we've already given, you must tell 
us your environment and show us some code.

Paul

[toc] | [prev] | [next] | [standalone]

#73387

On Wed, Jun 18, 2014 at 4:32 PM, cutey Love <cuteywithlove@gmail.com> wrote:
> Hi, thanks for the replies,
>
> I mean windows displays "Not responding close the program now"
>
> How can I do it asynconistrically?
>
> It's simple code just open file, loop through line by line and do some comparons with the string.

If all you want is to prevent Windows from displaying that message
then I believe all you need to do is periodically call
frame.update_idletasks() (or the equivalent if you're using some
framework other than tkinter) so that the basic window events will get
processed.

[toc] | [prev] | [next] | [standalone]

#73404

update_idletasks didn't work.

The code is this

    file_path = filedialog.askopenfilename(filetypes=[('text files', '.txt')], multiple=True, defaultextension=".txt")
    
    for path in file_path:
    
        fo = open(path, "r")
        
        for line in fo:
            if myCase(line.lower()):
                myList.append(line.lower())
        fo.close()
    

def myCase(c):
    
        if c in myList:
            return False
        
        if len(c) < 8 or len(c) > 80:
            return False

return True



This processes a fair bit of data





On Thursday, June 19, 2014 12:06:26 AM UTC+1, Ian wrote:
> On Wed, Jun 18, 2014 at 4:32 PM, cutey Love <cuteywithlove@gmail.com> wrote:
> 
> > Hi, thanks for the replies,
> 
> >
> 
> > I mean windows displays "Not responding close the program now"
> 
> >
> 
> > How can I do it asynconistrically?
> 
> >
> 
> > It's simple code just open file, loop through line by line and do some comparons with the string.
> 
> 
> 
> If all you want is to prevent Windows from displaying that message
> 
> then I believe all you need to do is periodically call
> 
> frame.update_idletasks() (or the equivalent if you're using some
> 
> framework other than tkinter) so that the basic window events will get
> 
> processed.

[toc] | [prev] | [next] | [standalone]

#73413

On 2014-06-19 09:17, cutey Love wrote:
> update_idletasks didn't work.
>
> The code is this
>
>      file_path = filedialog.askopenfilename(filetypes=[('text files', '.txt')], multiple=True, defaultextension=".txt")
>
>      for path in file_path:
>
>          fo = open(path, "r")
>
>          for line in fo:
>              if myCase(line.lower()):
>                  myList.append(line.lower())
>          fo.close()
>
>
> def myCase(c):
>
>          if c in myList:
>              return False
>
>          if len(c) < 8 or len(c) > 80:
>              return False
>
> return True
>
>
>
> This processes a fair bit of data
>
It's quicker to look for something in a set than in a list, so if you
can use a set instead of a list, do so.

Also, checking the length of a string is quick, quicker than searching
a list.

Therefore, before processing the file, do:

     mySet = set(myList)

and then you can say:

     def myCase(c):
         if len(c) < 8 or len(c) > 80:
             return False

         if c in mySet:
             return False

         return True

which can be shortened to:

     def myCase(c):
         return 8 <= len(c) <= 80 and c in mySet

[toc] | [prev] | [next] | [standalone]

#73437

On Thu, 19 Jun 2014 12:25:23 +0100, MRAB <python@mrabarnett.plus.com> wrote:
[snip]
> and then you can say:
>
>      def myCase(c):
>          if len(c) < 8 or len(c) > 80:
>              return False
>
>          if c in mySet:
>              return False
>
>          return True
>
> which can be shortened to:
>
>      def myCase(c):
>          return 8 <= len(c) <= 80 and c in mySet

Don't you mean . . .

        return 8 <= len(c) <= 80 and c not in mySet
?


-- 
To email me, substitute nowhere->spamcop, invalid->net.

[toc] | [prev] | [next] | [standalone]

#73438

On 2014-06-19 17:21, Peter Pearson wrote:
> On Thu, 19 Jun 2014 12:25:23 +0100, MRAB <python@mrabarnett.plus.com> wrote:
> [snip]
>> and then you can say:
>>
>>      def myCase(c):
>>          if len(c) < 8 or len(c) > 80:
>>              return False
>>
>>          if c in mySet:
>>              return False
>>
>>          return True
>>
>> which can be shortened to:
>>
>>      def myCase(c):
>>          return 8 <= len(c) <= 80 and c in mySet
>
> Don't you mean . . .
>
>          return 8 <= len(c) <= 80 and c not in mySet
> ?
>
Yes, you're right.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web