Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!1.eu.feeder.erje.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.003 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'else:': 0.03; 'duplicate': 0.07; 'intermediate': 0.07; 'problem:': 0.07; 'keys,': 0.09; 'rows': 0.09; 'rows,': 0.09; 'try:': 0.09; 'cc:addr:python-list': 0.11; '>>': 0.16; '-tkc': 0.16; '72k': 0.16; 'before.': 0.16; 'columns': 0.16; 'csv': 0.16; 'dict': 0.16; 'dictionaries': 0.16; 'does,': 0.16; 'exception:': 0.16; 'item[0]': 0.16; 'mylist': 0.16; 'stumbled': 0.16; 'to:addr:python.list': 0.16; 'to:addr:tim.thechases.com': 0.16; 'to:name:tim chase': 0.16; 'tuple.': 0.16; 'wrote:': 0.18; 'bit': 0.19; 'module': 0.19; 'file,': 0.19; '>>>': 0.22; 'coding': 0.22; 'cc:addr:python.org': 0.22; '>>>': 0.24; 'exists': 0.24; 'initial': 0.24; '(or': 0.24; 'cc:2**0': 0.24; 'cc:no real name:2**0': 0.24; '>': 0.26; 'header:In-Reply-To:1': 0.27; 'to:2**1': 0.27; 'tim': 0.29; 'message-id:@mail.gmail.com': 0.30; "i'm": 0.30; 'url:mailman': 0.30; 'work.': 0.31; 'code': 0.31; 'about.': 0.31; 'chase': 0.31; "d'aprano": 0.31; 'steven': 0.31; 'yesterday': 0.31; 'file': 0.32; 'this.': 0.32; 'url:python': 0.33; 'lab': 0.33; 'reader': 0.33; 'maybe': 0.34; 'except': 0.35; 'skip:s 30': 0.35; 'something': 0.35; 'received:google.com': 0.35; 'add': 0.35; 'really': 0.36; 'combination': 0.36; 'data,': 0.36; 'in.': 0.36; 'processed': 0.36; 'url:listinfo': 0.36; "i'll": 0.36; 'url:org': 0.36; 'should': 0.36; 'wrong': 0.37; 'so,': 0.37; 'list': 0.37; 'skip:o 20': 0.38; 'skip:& 10': 0.38; 'thank': 0.38; 'lists.': 0.38; 'pm,': 0.38; 'that,': 0.38; 'skip:& 20': 0.39; 'enough': 0.39; 'skip:p 20': 0.39; 'url:mail': 0.40; 'how': 0.40; 'remove': 0.60; 'staff': 0.61; "you're": 0.61; 'further': 0.61; '8bit%:10': 0.64; 'more': 0.64; 'accounts': 0.64; 'account': 0.65; 'hours': 0.66; 'reply': 0.66; 'dont': 0.67; '8bit%:21': 0.69; 'study': 0.69; 'clearer': 0.84; 'dict,': 0.84; 'approached': 0.93; 'skip:\xe4 10': 0.93; 'hundred': 0.95 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-type; bh=jseIi93474QOM0e/+XfhRx1n8fNX/TQnoQPxCnAJ2OU=; b=CvPpQaV8eUDbGTkU+Wu5sAli9pe2Z1MGnS8MBF5c3cNpKdYvBsu9Ynsb18fSheQKLr OjafNUYjrkgkkR63SweA0sOkxCEZbWJyK5PgK4aXbOuMCqDR+1yK9xLYvm6uahoQBMM9 bZTP40pPZU+0f2YWLVvhMfiOvVgYARQGwWfI9+U2XNdeIBsqxmSe/g959gJQwj5QUrJ9 mhLO3jmPSapWzBsEMhGcB3m9wdB4PxYzltZRwBOFVt5Z5AlHfMPm7GCwnsom0C2ggyd1 qX4Za/QearvgIPNehI28BduVRARd3qsOCmTaphA0cHMQnurWRjySKbQYbXsN23RNJzAp iG6w== X-Received: by 10.236.39.175 with SMTP id d35mr7646903yhb.119.1431660694933; Thu, 14 May 2015 20:31:34 -0700 (PDT) MIME-Version: 1.0 References: <5553f8fe$0$13012$c3e8da3$5496439d@news.astraweb.com> <5554D40C.9090505@pacbell.net> <20150514121759.73f98b76@bigbox.christie.dr> In-Reply-To: <20150514121759.73f98b76@bigbox.christie.dr> From: Ziqi Xiong Date: Fri, 15 May 2015 03:31:34 +0000 Subject: Re: Looking for direction To: Tim Chase , "20/20 Lab" Cc: python-list@python.org Content-Type: multipart/alternative; boundary=001a1138165829165a0516167b37 X-Mailman-Approved-At: Fri, 15 May 2015 09:28:46 +0200 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.20+ Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 205 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1431674927 news.xs4all.nl 2899 [2001:888:2000:d::a6]:41079 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:90654 --001a1138165829165a0516167b37 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable maybe we can change this list to dict, using item[0] and item[1] as keys, the whole item as value . then you can update by the same key i think Tim Chase =E4=BA=8E2015=E5=B9=B45=E6=9C=8815= =E6=97=A5 =E5=91=A8=E4=BA=9401:17=E5=86=99=E9=81=93=EF=BC=9A > On 2015-05-14 09:57, 20/20 Lab wrote: > > On 05/13/2015 06:23 PM, Steven D'Aprano wrote: > >>> I have a LARGE csv file that I need to process. 110+ columns, > >>> 72k rows. I managed to write enough to reduce it to a few > >>> hundred rows, and the five columns I'm interested in. > > I actually stumbled across the csv module after coding enough to > > make a list of lists. So that is more the reason I approached the > > list; Nothing like spending hours (or days) coding something that > > already exists and just dont know about. > >>> Now is were I have my problem: > >>> > >>> myList =3D [ [123, "XXX", "Item", "Qty", "Noise"], > >>> [72976, "YYY", "Item", "Qty", "Noise"], > >>> [123, "XXX" "ItemTypo", "Qty", "Noise"] ] > >>> > >>> Basically, I need to check for rows with duplicate accounts > >>> row[0] and staff (row[1]), and if so, remove that row, and add > >>> it's Qty to the original row. I really dont have a clue how to > >>> go about this. > >> > >> processed =3D {} # hold the processed data in a dict > >> > >> for row in myList: > >> account, staff =3D row[0:2] > >> key =3D (account, staff) # Put them in a tuple. > >> if key in processed: > >> # We've already seen this combination. > >> processed[key][3] +=3D row[3] # Add the quantities. > >> else: > >> # Never seen this combination before. > >> processed[key] =3D row > >> > >> newlist =3D list(processed.values()) > >> > > It does, immensely. I'll make this work. Thank you again for the > > link from yesterday and apologies for hitting the wrong reply > > button. I'll have to study more on the usage and implementations > > of dictionaries and tuples. > > In processing the initial CSV file, I suspect that using a > csv.DictReader would make the code a bit cleaner. Additionally, > as you're processing through the initial file, unless you need > the intermediate data, you should be able to do it in one pass. > Something like > > HEADER_ACCOUNT =3D "account" > HEADER_STAFF =3D "staff" > HEADER_QTY =3D "Qty" > > processed =3D {} > with open("data.csv") as f: > reader =3D csv.DictReader(f) > for row in reader: > if should_process_row(row): > account =3D row[HEADER_ACCOUNT] > staff =3D row[HEADER_STAFF] > qty =3D row[HEADER_QTY] > try: > row[HEADER_QTY] =3D qty =3D int(qty) > except Exception: > # not a numeric quantity? > continue > # from Steven's code > key =3D (account, staff) > if key in processed: > processed[key][HEADER_QTY] +=3D qty > else: > processed[key][HEADER_QTY] =3D row > so_something_with(processed.values()) > > I find that using names is a lot clearer than using arbitrary > indexing. Barring that, using indexes-as-constants still would > add further clarity. > > -tkc > > > > > . > -- > https://mail.python.org/mailman/listinfo/python-list > --001a1138165829165a0516167b37 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable maybe we can change this list to dict, using item[0] and item[1] as keys, t= he whole item as value . then you can update by the same key i think
Tim Chase <python.list@tim.thechases.com>=E4=BA=8E2015=E5=B9=B45=E6= =9C=8815=E6=97=A5 =E5=91=A8=E4=BA=9401:17=E5=86=99=E9=81=93=EF=BC=9A
On 2015-05-14 09:57, 20/20 Lab wrote:
> On 05/13/2015 06:23 PM, Steven D'Aprano wrote:
>>> I have a LARGE csv file that I need to process.=C2=A0 110+ col= umns,
>>> 72k rows.=C2=A0 I managed to write enough to reduce it to a fe= w
>>> hundred rows, and the five columns I'm interested in.
> I actually stumbled across the csv module after coding enough to
> make a list of lists.=C2=A0 So that is more the reason I approached th= e
> list; Nothing like spending hours (or days) coding something that
> already exists and just dont know about.
>>> Now is were I have my problem:
>>>
>>> myList =3D [ [123, "XXX", "Item", "Qt= y", "Noise"],
>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 [72976, "= YYY", "Item", "Qty", "Noise"],
>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 [123, "XX= X" "ItemTypo", "Qty", "Noise"]=C2=A0 =C2= =A0 ]
>>>
>>> Basically, I need to check for rows with duplicate accounts >>> row[0] and staff (row[1]), and if so, remove that row, and add=
>>> it's Qty to the original row. I really dont have a clue ho= w to
>>> go about this.
>>
>> processed =3D {}=C2=A0 # hold the processed data in a dict
>>
>> for row in myList:
>>=C2=A0 =C2=A0 =C2=A0 account, staff =3D row[0:2]
>>=C2=A0 =C2=A0 =C2=A0 key =3D (account, staff)=C2=A0 # Put them in a= tuple.
>>=C2=A0 =C2=A0 =C2=A0 if key in processed:
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # We've already seen this co= mbination.
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 processed[key][3] +=3D row[3]=C2= =A0 # Add the quantities.
>>=C2=A0 =C2=A0 =C2=A0 else:
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Never seen this combination be= fore.
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 processed[key] =3D row
>>
>> newlist =3D list(processed.values())
>>
> It does, immensely.=C2=A0 I'll make this work.=C2=A0 Thank you aga= in for the
> link from yesterday and apologies for hitting the wrong reply
> button.=C2=A0 I'll have to study more on the usage and implementat= ions
> of dictionaries and tuples.

In processing the initial CSV file, I suspect that using a
csv.DictReader would make the code a bit cleaner.=C2=A0 Additionally,
as you're processing through the initial file, unless you need
the intermediate data, you should be able to do it in one pass.
Something like

=C2=A0 HEADER_ACCOUNT =3D "account"
=C2=A0 HEADER_STAFF =3D "staff"
=C2=A0 HEADER_QTY =3D "Qty"

=C2=A0 processed =3D {}
=C2=A0 with open("data.csv") as f:
=C2=A0 =C2=A0 reader =3D csv.DictReader(f)
=C2=A0 =C2=A0 for row in reader:
=C2=A0 =C2=A0 =C2=A0 if should_process_row(row):
=C2=A0 =C2=A0 =C2=A0 =C2=A0 account =3D row[HEADER_ACCOUNT]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 staff =3D row[HEADER_STAFF]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 qty =3D row[HEADER_QTY]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 try:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 row[HEADER_QTY] =3D qty =3D int(qty)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 except Exception:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # not a numeric quantity?
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 continue
=C2=A0 =C2=A0 =C2=A0 =C2=A0 # from Steven's code
=C2=A0 =C2=A0 =C2=A0 =C2=A0 key =3D (account, staff)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if key in processed:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 processed[key][HEADER_QTY] +=3D qty
=C2=A0 =C2=A0 =C2=A0 =C2=A0 else:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 processed[key][HEADER_QTY] =3D row
=C2=A0 so_something_with(processed.values())

I find that using names is a lot clearer than using arbitrary
indexing.=C2=A0 Barring that, using indexes-as-constants still would
add further clarity.

-tkc




.
--
https://mail.python.org/mailman/listinfo/python-list
--001a1138165829165a0516167b37--