Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #44080

Re: itertools.groupby

Show all headers | View raw

On 22 April 2013 15:24, Neil Cerutti <neilc@norwich.edu> wrote:
>
> Hrmmm, hoomm. Nobody cares for slicing any more.
>
> def headered_groups(lst, header):
>     b = lst.index(header) + 1
>     while True:
>         try:
>             e = lst.index(header, b)
>         except ValueError:
>             yield lst[b:]
>             break
>         yield lst[b:e]
>         b = e+1

This requires the whole file to be read into memory. Iterators are
typically preferred over list slicing for sequential text file access
since you can avoid loading the whole file at once. This means that
you can process a large file while only using a constant amount of
memory.

>
> for group in headered_groups([line.strip() for line in open('data.txt')],
>         "Starting a new group"):
>     print(group)

The list comprehension above loads the entire file into memory.
Assuming that .strip() is just being used to remove the newline at the
end it would be better to just use the readlines() method since that
loads everything into memory and removes the newlines. To remove them
without reading everything you can use map (or imap in Python 2):

with open('data.txt') as inputfile:
    for group in headered_groups(map(str.strip, inputfile)):
        print(group)


Oscar

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

itertools.groupby Jason Friedman <jsf80238@gmail.com> - 2013-04-20 11:09 -0600
  Re: itertools.groupby Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-21 00:13 +0000
    Re: itertools.groupby Joshua Landau <joshua.landau.ws@gmail.com> - 2013-04-22 04:09 +0100
  Re: itertools.groupby Neil Cerutti <neilc@norwich.edu> - 2013-04-22 14:24 +0000
    Re: itertools.groupby Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-22 15:49 +0100
      Re: itertools.groupby Neil Cerutti <neilc@norwich.edu> - 2013-04-22 15:04 +0000
    Re: itertools.groupby Chris Angelico <rosuav@gmail.com> - 2013-04-23 01:14 +1000

csiph-web