Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #11279

Re: Processing a large string

Show all headers | View raw

Peter Otten wrote:

> goldtech wrote:

>> Say I have a very big string with a pattern like:
>> 
>> akakksssk3dhdhdhdbddb3dkdkdkddk3dmdmdmd3dkdkdkdk3asnsn.....
>> 
>> I want to split the sting into separate parts on the "3" and process
>> each part separately. I might run into memory limitations if I use
>> "split" and get a big array(?)  I wondered if there's a way I could
>> read (stream?) the string from start to finish and read what's
>> delimited by the "3" into a variable, process the smaller string
>> variable then append/build a new string with the processed data?

> PS: This has come up before, but I couldn't find the relevant threads...

Alex Martelli a looong time ago:

> from __future__ import generators
> 
> def splitby(fileobj, splitter, bufsize=8192):
>     buf = ''
> 
>     while True:
>         try: 
>             item, buf = buf.split(splitter, 1)
>         except ValueError:
>             more = fileobj.read(bufsize)
>             if not more: break
>             buf += more
>         else:
>             yield item + splitter
> 
>     if buf:
>         yield buf

http://mail.python.org/pipermail/python-list/2002-September/770673.html

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Processing a large string goldtech <goldtech@worldpost.com> - 2011-08-11 19:03 -0700
  Re: Processing a large string MRAB <python@mrabarnett.plus.com> - 2011-08-12 03:15 +0100
  Re: Processing a large string Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-08-12 12:30 +1000
  Re: Processing a large string Nobody <nobody@nowhere.com> - 2011-08-12 05:11 +0100
  Re: Processing a large string Peter Otten <__peter__@web.de> - 2011-08-12 10:39 +0200
    Re: Processing a large string goldtech <goldtech@worldpost.com> - 2011-08-12 06:36 -0700
    Re: Processing a large string Peter Otten <__peter__@web.de> - 2011-08-12 16:48 +0200
  Re: Processing a large string Paul Rudin <paul.nospam@rudin.co.uk> - 2011-08-28 20:18 +0100

csiph-web