Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #78444 > unrolled thread

trouble building data structure

Started byDavid Alban <extasia@extasia.org>
First post2014-09-28 17:04 -0700
Last post2014-09-29 09:36 -0600
Articles 3 — 3 participants

Back to article view | Back to comp.lang.python


Contents

  trouble building data structure David Alban <extasia@extasia.org> - 2014-09-28 17:04 -0700
    Re: trouble building data structure Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-09-29 23:52 +1000
      Re: trouble building data structure Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-29 09:36 -0600

#78444 — trouble building data structure

FromDavid Alban <extasia@extasia.org>
Date2014-09-28 17:04 -0700
Subjecttrouble building data structure
Message-ID<mailman.14456.1411949456.18130.python-list@python.org>

[Multipart message — attachments visible in raw view] — view raw

greetings,

i'm writing a program to scan a data file.  from each line of the data file
i'd like to add something like below to a dictionary.  my perl background
makes me want python to autovivify, but when i do:

      file_data = {}

      [... as i loop through lines in the file ...]

          file_data[ md5sum ][ inode ] = { 'path' : path, 'size' : size, }

i get:

Traceback (most recent call last):
  File "foo.py", line 45, in <module>
    file_data[ md5sum ][ inode ] = { 'path' : path, 'size' : size, }
KeyError: '91b152ce64af8af91dfe275575a20489'

what is the pythonic way to build my "file_data" data structure above that
has the above structure?

on http://en.wikipedia.org/wiki/Autovivification there is a section on how
to do autovivification in python, but i want to learn how a python
programmer would normally build a data structure like this.

here is the code so far:

#!/usr/bin/python

import argparse
import os

ASCII_NUL = chr(0)

HOSTNAME = 0
MD5SUM   = 1
FSDEV    = 2
INODE    = 3
NLINKS   = 4
SIZE     = 5
PATH     = 6

file_data = {}

if __name__ == "__main__":
  parser = argparse.ArgumentParser(description='scan files in a tree and
print a line of information about each regular file')
  parser.add_argument('--file', '-f', required=True, help='File from which
to read data')
  parser.add_argument('--field-separator', '-s', default=ASCII_NUL,
help='Specify the string to use as a field separator in output.  The
default is the ascii nul character.')
  args = parser.parse_args()

  file = args.file
  field_separator = args.field_separator

  with open( file, 'rb' ) as f:
    for line in f:
      line = line.rstrip('\n')
      if line == 'None': continue
      fields = line.split( ASCII_NUL )

      hostname = fields[ HOSTNAME ]
      md5sum   = fields[ MD5SUM ]
      fsdev    = fields[ FSDEV ]
      inode    = fields[ INODE ]
      nlinks   = int( fields[ NLINKS ] )
      size     = int( fields[ SIZE ] )
      path     = fields[ PATH ]

      if size < ( 100 * 1024 * 1024 ): continue

      ### print "'%s' '%s' '%s' '%s' '%s' '%s' '%s'" % ( hostname, md5sum,
fsdev, inode, nlinks, size, path, )

      file_data[ md5sum ][ inode ] = { 'path' : path, 'size' : size, }

thanks,
david
-- 
Our decisions are the most important things in our lives.
***
Live in a world of your own, but always welcome visitors.

[toc] | [next] | [standalone]


#78464

FromSteven D'Aprano <steve+comp.lang.python@pearwood.info>
Date2014-09-29 23:52 +1000
Message-ID<54296401$0$12999$c3e8da3$5496439d@news.astraweb.com>
In reply to#78444
David Alban wrote:

> greetings,
> 
> i'm writing a program to scan a data file.  from each line of the data
> file
> i'd like to add something like below to a dictionary.  my perl background
> makes me want python to autovivify, but when i do:
> 
>       file_data = {}
> 
>       [... as i loop through lines in the file ...]
> 
>           file_data[ md5sum ][ inode ] = { 'path' : path, 'size' : size, }
> 
> i get:
> 
> Traceback (most recent call last):
>   File "foo.py", line 45, in <module>
>     file_data[ md5sum ][ inode ] = { 'path' : path, 'size' : size, }
> KeyError: '91b152ce64af8af91dfe275575a20489'
> 
> what is the pythonic way to build my "file_data" data structure above that
> has the above structure?

Others have suggested using a defaultdict, but here's an older solution: use
the setdefault method on regular dicts.

This fails:

py> file_data = {}
py> file_data[1234][23] = {'spam': 1, 'eggs': 2}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 1234


But this succeeds:

py> file_data.setdefault(1234, {})[23] = {'spam': 1, 'eggs': 2}
py> file_data.setdefault(5678, {})[42] = {'spam': 3, 'cheese': 1}
py> file_data
{1234: {23: {'spam': 1, 'eggs': 2}}, 5678: {42: {'spam': 3, 'cheese': 1}}}

Whether you prefer to use setdefault, or a defaultdict, is a matter of
taste.


-- 
Steven

[toc] | [prev] | [next] | [standalone]


#78470

FromIan Kelly <ian.g.kelly@gmail.com>
Date2014-09-29 09:36 -0600
Message-ID<mailman.14474.1412005035.18130.python-list@python.org>
In reply to#78464
On Mon, Sep 29, 2014 at 7:52 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Whether you prefer to use setdefault, or a defaultdict, is a matter of
> taste.

There is potentially a significant difference in performance -- with
setdefault, the subordinate data structure is created on every call to
be passed into setdefault, only to be discarded if the key already
exists. With defaultdict, the subordinate data structure is only
created when needed. Dicts are pretty cheap to construct though, and
this is probably not worth fretting over until profiling shows it to
be a problem.

On the other hand, it's easier to nest setdefaults arbitrarily deep
than it is for defaultdicts. This is because defaultdict suffers from
a design flaw -- defaultdict should be a function that returns a class
(like namedtuple), not a class itself. Fortunately that's easily fixable:

_DEFAULT_DICT_CACHE = {}

def defaultdict(callable):
  try:
    return _DEFAULT_DICT_CACHE[callable]
  except KeyError:
    class _defaultdict(dict):
      def __missing__(self, key):
        self[key] = value = self._callable()
        return value

    _DEFAULT_DICT_CACHE[callable] = _defaultdict
    return _defaultdict

A downside is that it would take some extra work to make this picklable.

[toc] | [prev] | [standalone]


Back to top | Article view | comp.lang.python


csiph-web