Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #73413

Re: Not Responding When Dealing with Large Data

Show key headers only | View raw

On 2014-06-19 09:17, cutey Love wrote:
> update_idletasks didn't work.
>
> The code is this
>
>      file_path = filedialog.askopenfilename(filetypes=[('text files', '.txt')], multiple=True, defaultextension=".txt")
>
>      for path in file_path:
>
>          fo = open(path, "r")
>
>          for line in fo:
>              if myCase(line.lower()):
>                  myList.append(line.lower())
>          fo.close()
>
>
> def myCase(c):
>
>          if c in myList:
>              return False
>
>          if len(c) < 8 or len(c) > 80:
>              return False
>
> return True
>
>
>
> This processes a fair bit of data
>
It's quicker to look for something in a set than in a list, so if you
can use a set instead of a list, do so.

Also, checking the length of a string is quick, quicker than searching
a list.

Therefore, before processing the file, do:

     mySet = set(myList)

and then you can say:

     def myCase(c):
         if len(c) < 8 or len(c) > 80:
             return False

         if c in mySet:
             return False

         return True

which can be shortened to:

     def myCase(c):
         return 8 <= len(c) <= 80 and c in mySet

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Not Responding When Dealing with Large Data cutey Love <cuteywithlove@gmail.com> - 2014-06-18 10:20 -0700
  Re: Not Responding When Dealing with Large Data Philip Dexter <philip.dexter@gmail.com> - 2014-06-18 13:36 -0400
  Re: Not Responding When Dealing with Large Data Tim <jtim.arnold@gmail.com> - 2014-06-18 11:37 -0700
  Re: Not Responding When Dealing with Large Data John Gordon <gordon@panix.com> - 2014-06-18 19:10 +0000
  Re: Not Responding When Dealing with Large Data Michael Torrie <torriem@gmail.com> - 2014-06-18 13:25 -0600
  Re: Not Responding When Dealing with Large Data Paul McNett <paul@mcnettware.com> - 2014-06-18 12:23 -0700
  Re: Not Responding When Dealing with Large Data cutey Love <cuteywithlove@gmail.com> - 2014-06-18 15:32 -0700
    Re: Not Responding When Dealing with Large Data Paul McNett <paul@mcnettware.com> - 2014-06-18 15:48 -0700
    Re: Not Responding When Dealing with Large Data Ian Kelly <ian.g.kelly@gmail.com> - 2014-06-18 17:06 -0600
      Re: Not Responding When Dealing with Large Data cutey Love <cuteywithlove@gmail.com> - 2014-06-19 01:17 -0700
        Re: Not Responding When Dealing with Large Data MRAB <python@mrabarnett.plus.com> - 2014-06-19 12:25 +0100
          Re: Not Responding When Dealing with Large Data Peter Pearson <ppearson@nowhere.invalid> - 2014-06-19 16:21 +0000
            Re: Not Responding When Dealing with Large Data MRAB <python@mrabarnett.plus.com> - 2014-06-19 17:54 +0100

csiph-web