Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #75404

Re: speed up pandas calculation

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

On Wed, Jul 30, 2014 at 5:57 PM, Skip Montanaro <skip.montanaro@gmail.com>
wrote:

> > df = pd.read_csv('nhamcsopd2010.csv' , index_col='PATCODE',
> low_memory=False)
> > col_init = list(df.columns.values)
> > keep_col = ['PATCODE', 'PATWT', 'VDAY', 'VMONTH', 'VYEAR', 'MED1',
> 'MED2', 'MED3', 'MED4', 'MED5']
> > for col in col_init:
> >     if col not in keep_col:
> >         del df[col]
>
> I'm no pandas expert, but a couple things come to mind. First, where is
> your code slow (profile it, even with a few well-placed prints)? If it's in
> read_csv there might be little you can do unless you load those data
> repeatedly, and can save a pickled data frame as a caching measure. Second,
> you loop over columns deciding one by one whether to keep or toss a column.
> Instead try
>
> df = df[keep_col]
>
> Third, if deleting those other columns is costly, can you perhaps just
> ignore them?
>
> Can't be more investigative right now. I don't have pandas on Android. :-)
>

So the df = df[keep_col] is not fast but it is not that slow. You made me
think of a solution to that part. just slice and copy. The only gotya is
that the keep_col have to actually exist
 keep_col = ['PATCODE', 'PATWT', 'VDAYR', 'VMONTH', 'MED1', 'MED2', 'MED3',
'MED4', 'MED5']
df = df[keep_col]

The real slow part seems to be
for n in drugs:
    df[n] = df[['MED1','MED2','MED3','MED4','MED5']].isin([drugs[n]]).any(1)



Vincent Davis
720-301-3003

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: speed up pandas calculation Vincent Davis <vincent@vincentdavis.net> - 2014-07-30 18:28 -0600

csiph-web