Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #101317

Re: Fast pythonic way to process a huge integer list

Show all headers | View raw

On 2016-01-06 18:36, high5storage@gmail.com wrote:
> I have a list of 163.840 integers. What is a fast & pythonic way to
> process this list in 1,280 chunks of 128 integers?

That's a modest list, far from huge.

You have lots of options, but the following seems the most pythonic to
me:

  # I don't know how you populate your data so
  # create some junk data
  from random import randint
  data = [randint(0,1000) for _ in range(163840)]

  import itertools as i
  GROUP_SIZE = 128

  def do_something(grp, vals):
    for _, val in vals:
      # I don't know what you want to do with each
      # pair.  You can print them:

      # print("%s: %s" % (grp, val))

      # or write them to various chunked files:
      with open("chunk%04i.txt" % grp, "w") as f:
        f.write(str(val))
        f.write("\n")

  # but here's the core logic:

  def key_fn(x):
    # x is a tuple of (index, value)
    return x[0] // GROUP_SIZE

  # actually iterate over the grouped data
  # and do something with it:
  for grp, vals in i.groupby(enumerate(data), key_fn):
    do_something(grp, vals)

-tkc

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Fast pythonic way to process a huge integer list high5storage@gmail.com - 2016-01-06 18:36 -0800
  Re: Fast pythonic way to process a huge integer list Terry Reedy <tjreedy@udel.edu> - 2016-01-06 22:10 -0500
  Re: Fast pythonic way to process a huge integer list Tim Chase <python.list@tim.thechases.com> - 2016-01-06 21:21 -0600
  Re: Fast pythonic way to process a huge integer list Cameron Simpson <cs@zip.com.au> - 2016-01-07 14:31 +1100
  Re: Fast pythonic way to process a huge integer list Steven D'Aprano <steve@pearwood.info> - 2016-01-07 20:25 +1100
  Re: Fast pythonic way to process a huge integer list Peter Otten <__peter__@web.de> - 2016-01-07 11:21 +0100
  Re: Fast pythonic way to process a huge integer list KP <kai.peters@gmail.com> - 2016-01-07 16:33 -0800

csiph-web