Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #10693

Re: python reading file memory cost

Show key headers only | View raw

Thanks!

Actually, I used .readline() to parse file line by line, because I need
to find out the start position to extract data into list, and the end
point to pause extracting, then repeat until the end of file.
My file to read is formatted like this:

blabla...useless....
useless...

/sign/
data block(e.g. 10 cols x 1000 rows)
...
blank line
/sign/
data block(e.g. 10 cols x 1000 rows)
...
blank line
...
...
EOF
let's call this file 'myfile'
and my python snippet:

f=open('myfile','r')
blocknum=0 #number the data block
data=[]
while True"
	# find the extract begnning
	while not f.readline().startswith('/a1/'):pass
	# creat multidimensional list to store data block	
	data=append([])
	blocknum +=1
	line=f.readline()

	while line.strip():  
	# check if the line is a blank line, i.e the end of one block
		data[blocknum-1].append(["2.6E" %float(x) for x in line.split()])
		line = f.readline()
	print "Read Block %d" %blocknum
	if not f.readline(): break 

The running result was that read a 500M file consume almost 2GB RAM, I
cannot figure it out, somebody help!
Thanks very much!

--Tony

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: python reading file memory cost Tony Zhang <warriorlance@gmail.com> - 2011-08-02 11:22 +0800

csiph-web