Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #43343

Re: performance of script to write very long lines of random chars

Show all headers | View raw

On 11 April 2013 02:21, gry <georgeryoung@gmail.com> wrote:
> Dear pythonistas,
>    I am writing a tiny utility to produce a file consisting of a
> specified number of lines of a given length of random ascii
> characters.  I am hoping to find a more time and memory efficient way,
> that is still fairly simple clear, and _pythonic_.
>
> I would like to have something that I can use at both extremes of
> data:
>
>    32M chars per line * 100 lines
> or
>    5 chars per line * 1e8 lines.

I would definitely use numpy for this. The script below seems to be
io-bound on my machine:

#!/usr/bin/env python

from sys import stdout
import numpy as np
from numpy import random

CHARS = (
    '0123456789'
    'abcdefghijklmnopqrstuvwxyz'
    'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    '!#$%& \'()*+,-./:;<=>?@[\\]^_`{}'
    )

ARRAY_CHARS = np.frombuffer(CHARS, np.uint8)
NCHARS = len(CHARS)

CHUNK_SIZE = 4096
NCOLS = 32000000
NROWS = 10

def chunk_sizes(total, chunk_size):
    numchunks, remainder = divmod(total, chunk_size)
    for n in range(numchunks):
        yield chunk_size
    if remainder:
        yield remainder

def chunks():
    bytes_per_line = NCOLS + 1
    total_bytes = bytes_per_line * NROWS
    newline_index = NCOLS
    newline = ord('\n')
    for size in chunk_sizes(total_bytes, CHUNK_SIZE):
        chars = ARRAY_CHARS[random.randint(0, NCHARS, size)]
        chars[newline_index::bytes_per_line] = newline
        newline_index = (newline_index - CHUNK_SIZE) % bytes_per_line
        yield chars

for chunk in chunks():
    chunk.tofile(stdout)

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

performance of script to write very long lines of random chars gry <georgeryoung@gmail.com> - 2013-04-10 18:21 -0700
  Re: performance of script to write very long lines of random chars Chris Angelico <rosuav@gmail.com> - 2013-04-11 11:45 +1000
    Re: performance of script to write very long lines of random chars Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-11 05:33 +0000
      Re: performance of script to write very long lines of random chars Chris Angelico <rosuav@gmail.com> - 2013-04-11 15:53 +1000
  Re: performance of script to write very long lines of random chars Michael Torrie <torriem@gmail.com> - 2013-04-10 19:52 -0600
    Re: performance of script to write very long lines of random chars gry <georgeryoung@gmail.com> - 2013-04-10 19:40 -0700
      Re: performance of script to write very long lines of random chars Chris Angelico <rosuav@gmail.com> - 2013-04-11 13:14 +1000
  Re: performance of script to write very long lines of random chars MRAB <python@mrabarnett.plus.com> - 2013-04-11 04:09 +0100
  Re: performance of script to write very long lines of random chars Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-11 07:47 +0000
    Re: performance of script to write very long lines of random chars Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-11 10:47 +0100
      Re: performance of script to write very long lines of random chars Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-11 10:50 +0000
        Re: performance of script to write very long lines of random chars Robert Kern <robert.kern@gmail.com> - 2013-04-11 16:49 +0530
        Re: performance of script to write very long lines of random chars Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-11 13:05 +0100
        Re: performance of script to write very long lines of random chars Robert Kern <robert.kern@gmail.com> - 2013-04-11 19:06 +0530
        Re: performance of script to write very long lines of random chars Chris Angelico <rosuav@gmail.com> - 2013-04-11 23:56 +1000
  Re: performance of script to write very long lines of random chars Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-11 10:47 +0100

csiph-web