Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!newsreader4.netcologne.de!news.netcologne.de!bcyclone04.am1.xlned.com!bcyclone04.am1.xlned.com!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Date: Thu, 14 May 2015 01:41:23 +0100
From: MRAB <python@mrabarnett.plus.com>
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Looking for direction
References: <5553DD2E.2080600@pacbell.net> <5553E71F.8090402@stoneleaf.us>
In-Reply-To: <5553E71F.8090402@stoneleaf.us>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.471.1431564092.12865.python-list@python.org>
Lines: 69
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:90595

On 2015-05-14 01:06, Ethan Furman wrote:
> On 05/13/2015 04:24 PM, 20/20 Lab wrote:
>> I'm a beginner to python.  Reading here and there.  Written a couple of
>> short and simple programs to make life easier around the office.
>>
>> That being said, I'm not even sure what I need to ask for. I've never
>> worked with external data before.
>>
>> I have a LARGE csv file that I need to process.  110+ columns, 72k
>> rows.  I managed to write enough to reduce it to a few hundred rows, and
>> the five columns I'm interested in.
>>
>> Now is were I have my problem:
>>
>> myList = [ [123, "XXX", "Item", "Qty", "Noise"],
>>             [72976, "YYY", "Item", "Qty", "Noise"],
>>             [123, "XXX" "ItemTypo", "Qty", "Noise"]    ]
>>
>> Basically, I need to check for rows with duplicate accounts row[0] and
>> staff (row[1]), and if so, remove that row, and add it's Qty to the
>> original row. I really dont have a clue how to go about this.  The
>> number of rows change based on which run it is, so I couldnt even get
>> away with using hundreds of compare loops.
>>
>> If someone could point me to some documentation on the functions I would
>> need, or a tutorial it would be a great help.
>
> You could try using a dictionary, combining when needed:
>
> # untested
> data = {}
> for row in all_rows:
>     key = row[0], row[1]
>     if key in data:
>       item, qty, noise = data[key]
>       qty += row[3]
>     else:
>       item, qty, noise = row[2:]
>     data[key] = item, qty, noise
>
> for (account, staff), (item, qty, noise) in data.items():
>     do_stuff_with(account, staff, item, qty, noise)
>
> At the end, data should have what you want.  It won't, however, be in
> the same order, so hopefully that's not an issue for you.
>
Starting from that, if the order matters, you can do it this way:

data = {}
order = {}
for index, row in enumerate(all_rows):
     key = row[0], row[1]
     if key in data:
         item, qty, noise = data[key]
         qty += row[3]
     else:
         item, qty, noise = row[2:]
     data[key] = item, qty, noise
     order.setdefault(key, index)

merged_rows = [(account, staff, item, qty, noise) for (account, staff), 
(item, qty, noise) in data.items()]

def original_order(row):
     key = row[0], row[1]
     return order[key]

merged_rows.sort(key=original_order)