Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Thu, 14 May 2015 12:17:59 -0500
From: Tim Chase <python.list@tim.thechases.com>
To: 20/20 Lab <lab@pacbell.net>
Cc: python-list@python.org
Subject: Re: Looking for direction
In-Reply-To: <5554D40C.9090505@pacbell.net>
References: <mailman.465.1431559626.12865.python-list@python.org> <5553f8fe$0$13012$c3e8da3$5496439d@news.astraweb.com> <5554D40C.9090505@pacbell.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.9.1431623856.17265.python-list@python.org>
Lines: 80
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:90618

On 2015-05-14 09:57, 20/20 Lab wrote:
> On 05/13/2015 06:23 PM, Steven D'Aprano wrote:
>>> I have a LARGE csv file that I need to process.  110+ columns,
>>> 72k rows.  I managed to write enough to reduce it to a few
>>> hundred rows, and the five columns I'm interested in.
> I actually stumbled across the csv module after coding enough to
> make a list of lists.  So that is more the reason I approached the
> list; Nothing like spending hours (or days) coding something that
> already exists and just dont know about.
>>> Now is were I have my problem:
>>>
>>> myList = [ [123, "XXX", "Item", "Qty", "Noise"],
>>>              [72976, "YYY", "Item", "Qty", "Noise"],
>>>              [123, "XXX" "ItemTypo", "Qty", "Noise"]    ]
>>>
>>> Basically, I need to check for rows with duplicate accounts
>>> row[0] and staff (row[1]), and if so, remove that row, and add
>>> it's Qty to the original row. I really dont have a clue how to
>>> go about this.
>>
>> processed = {}  # hold the processed data in a dict
>>
>> for row in myList:
>>      account, staff = row[0:2]
>>      key = (account, staff)  # Put them in a tuple.
>>      if key in processed:
>>          # We've already seen this combination.
>>          processed[key][3] += row[3]  # Add the quantities.
>>      else:
>>          # Never seen this combination before.
>>          processed[key] = row
>>
>> newlist = list(processed.values())
>>
> It does, immensely.  I'll make this work.  Thank you again for the
> link from yesterday and apologies for hitting the wrong reply
> button.  I'll have to study more on the usage and implementations
> of dictionaries and tuples.

In processing the initial CSV file, I suspect that using a
csv.DictReader would make the code a bit cleaner.  Additionally,
as you're processing through the initial file, unless you need
the intermediate data, you should be able to do it in one pass.
Something like

  HEADER_ACCOUNT = "account"
  HEADER_STAFF = "staff"
  HEADER_QTY = "Qty"

  processed = {}
  with open("data.csv") as f:
    reader = csv.DictReader(f)
    for row in reader:
      if should_process_row(row):
        account = row[HEADER_ACCOUNT]
        staff = row[HEADER_STAFF]
        qty = row[HEADER_QTY]
        try:
          row[HEADER_QTY] = qty = int(qty)
        except Exception:
          # not a numeric quantity?
          continue
        # from Steven's code
        key = (account, staff)
        if key in processed:
          processed[key][HEADER_QTY] += qty
        else:
          processed[key][HEADER_QTY] = row
  so_something_with(processed.values())
          
I find that using names is a lot clearer than using arbitrary
indexing.  Barring that, using indexes-as-constants still would
add further clarity.

-tkc




.