Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #90613

Re: Looking for direction

Show key headers only | View raw

On 05/13/2015 06:23 PM, Steven D'Aprano wrote:
> On Thu, 14 May 2015 09:24 am, 20/20 Lab wrote:
>
>> I'm a beginner to python.  Reading here and there.  Written a couple of
>> short and simple programs to make life easier around the office.
>>
>> That being said, I'm not even sure what I need to ask for. I've never
>> worked with external data before.
>>
>> I have a LARGE csv file that I need to process.  110+ columns, 72k
>> rows.  I managed to write enough to reduce it to a few hundred rows, and
>> the five columns I'm interested in.
> That's not large. Large is millions of rows, or tens of millions if you have
> enough memory. What's large to you and me is usually small to the computer.
>
> You should use the csv module for handling the CSV file, if you aren't
> already doing so. Do you need a url to the docs?
>
I actually stumbled across the csv module after coding enough to make a 
list of lists.  So that is more the reason I approached the list;  
Nothing like spending hours (or days) coding something that already 
exists and just dont know about.
>> Now is were I have my problem:
>>
>> myList = [ [123, "XXX", "Item", "Qty", "Noise"],
>>              [72976, "YYY", "Item", "Qty", "Noise"],
>>              [123, "XXX" "ItemTypo", "Qty", "Noise"]    ]
>>
>> Basically, I need to check for rows with duplicate accounts row[0] and
>> staff (row[1]), and if so, remove that row, and add it's Qty to the
>> original row. I really dont have a clue how to go about this.
> Is the order of the rows important? If not, the problem is simpler.
>
>
> processed = {}  # hold the processed data in a dict
>
> for row in myList:
>      account, staff = row[0:2]
>      key = (account, staff)  # Put them in a tuple.
>      if key in processed:
>          # We've already seen this combination.
>          processed[key][3] += row[3]  # Add the quantities.
>      else:
>          # Never seen this combination before.
>          processed[key] = row
>
> newlist = list(processed.values())
>
>
> Does that help?
>
>
>
It does, immensely.  I'll make this work.  Thank you again for the link 
from yesterday and apologies for hitting the wrong reply button.  I'll 
have to study more on the usage and implementations of dictionaries and 
tuples.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Looking for direction 20/20 Lab <lab@pacbell.net> - 2015-05-13 16:24 -0700
  Re: Looking for direction Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-14 11:23 +1000
    Re: Looking for direction 20/20 Lab <lab@pacbell.net> - 2015-05-14 09:57 -0700
    Re: Looking for direction Tim Chase <python.list@tim.thechases.com> - 2015-05-14 12:17 -0500
    Re: Looking for direction Ziqi Xiong <xiongziqi84@gmail.com> - 2015-05-15 03:31 +0000
  Re: Looking for direction darnold <darnold992000@yahoo.com> - 2015-05-20 05:50 -0700
    Re: Looking for direction 20/20 Lab <lab@pacbell.net> - 2015-05-20 14:18 -0700

csiph-web