Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #9040

Re: find max and min values from a column of a csv file

Show all headers | View raw

On Thu, 7 Jul 2011 15:28:25 +0530, prakash jp <prakash.stack@gmail.com>
declaimed the following in gmane.comp.python.general:

> Hi All ,
> 
> Could any one help to get max and min values from a specified column of a
> csv file. The* csv file is large* and hence the below code did go bad.

	"did go bad", HOW?

	Off-hand, it looks obscenely slow with lots of redundant actions.


> *Alternate
> methods would be highly appreciated
> **
> minimum.py*:
> import csv
> filename = "testLog_4.csv"
> f = open(filename)

	You opened the file as "f"

> def col_min(mincname):
>     with open(filename, 'rb') as f:

	You just REOPENED the file in binary mode, as "f"

>         reader = csv.reader(f, delimiter=",")
>         #print reader
>         #print mincname
>         col_index = next(reader).index(mincname)
>         #print col_index
>         least = min(rec[col_index] for rec in reader)

	You read the entire file to collect just one column of data and
compute a min on it.

	The "with" block exits, and "f" is closed (that's one of the
purposes of the "with" statement)

>     print least
> 
> col_min(str("Server Latency"))
> col_min(str("Client Latency"))

	You've just done TWO passes over the file.

> f.close()

	And closed a file that is already closed... Unless "with" creates a
nested scope so its "f" is a different "f"

> 
> *maximum.py:*
> import csv
> filename = "testLog_4.csv"
> f = open(filename)
> def col_max(maxcname):
>     with open(filename, 'rb') as f:
>         reader = csv.reader(f, delimiter=",")
>         #print reader
>         #print cname
>         col_index = next(reader).index(maxcname)
>         #print col_index
>         highest = max(rec[col_index] for rec in reader)
>     print highest
> 
> col_max(str("Server Latency"))
> col_max(str("Client Latency"))
> f.close()

	And here you repeat the operation using max and two more passes over
the file.

	Side comment, you only need one function to do the process part...
Not two...

def collectData(colName, function):
	...
	return function(rec[col_index] for rec in reader)

col_max = collectData("Server Latency", max)
col_min = collectData("Server Latency", min)

	But fastest should be to do just one pass over the data

UNTESTED
-=-=-=-=-=-=-
import csv

DATAFILE = "testLog_4.csv"

c_min = 9999999999999999999L    #I don't know the domain range, pick a
suitable
s_min = 9999999999999999999L
c_max = -9999999999999999999L
s_max = -9999999999999999999L

fin = open(DATAFILE, "rb")
rdr = csv.reader(fin)

#   NO ERROR TRAPPING

header = rdr.next()
clientIndex = header.index("Client Latency")
serverIndex = header.index("Server Latency")

for record in rdr:
    client = long(record[clientIndex])  #use appropriate data type
    server = long(record[serverIndex])
    c_min = min(c_min, client)
    c_max = max(c_max, client)
    s_min = min(s_min, server)
    s_max = max(s_max, server)

print "Server Latency min/max: %s/%s" % (s_min, s_max)
print "Client Latency min/max: %s/%s" % (c_min, c_max)

fin.close()
del rdr
-=-=-=-=-=-=-
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: find max and min values from a column of a csv file Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2011-07-07 10:18 -0700

csiph-web