Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #4776

avro slow?

Show all headers | View raw

Greetings,

I'm reading some data from avro file using the avro library. It takes about a minute to load 33K objects from the file. This seem very slow to me, specially with the Java version reading the same file in about 1sec.

Here is the code, am I doing something wrong?

    import avro.datafile
    import avro.io
    from time import time

    def load(filename):
        fo = open(filename, "rb")
        reader = avro.datafile.DataFileReader(fo, avro.io.DatumReader())
        for i, record in enumerate(reader):
            pass

        return i + 1

    def main(argv=None):
        import sys
        from argparse import ArgumentParser

        argv = argv or sys.argv

        parser = ArgumentParser(description="Read avro file")


        start = time()
        num_records = load("events.avro")
        end = time()

        print("{0} records in {1} seconds".format(num_records, end - start))

    if __name__ == "__main__":
        main()

Back to comp.lang.python | Previous | Next — Next in thread | Find similar

Thread

avro slow? Miki Tebeka <miki.tebeka@gmail.com> - 2011-05-05 14:12 -0700
  Re: avro slow? Stefan Behnel <stefan_ml@behnel.de> - 2011-05-06 01:18 +0200

csiph-web