Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #31787

Re: Python does not take up available physical memory

Show all headers | View raw

On Fri, 19 Oct 2012 17:20:23 -0500, Pradipto Banerjee
<pradipto.banerjee@adainvestments.com> declaimed the following in
gmane.comp.python.general:

> Thanks, for the illustration. This seems to be one of the biggest shortcomings of Python vs. Matlab. A number of people told me to read one line at a time, but I have a need to run processes on the whole data, e.g. compare one line versus another. So that option doesn't work.

	And that requirement already suggests that reading the file en-mass
is inappropriate... Reading a 1GB mass and THEN splitting it into lines
means you have 2GB (not counting overhead) in memory for some period of
time (assuming the OS found a 1GB contiguous chunk of memory).

	I suspect Matlab's read is internally parsing on lines. You don't
show the related Matlab read statement but...
http://www.mathworks.com/help/matlab/ref/fscanf.html does both the read
AND the conversion to the binary array format -- it doesn't read the
file as a chunk and THEN convert it to an array; it only reads enough to
fulfill one "format" string, saves that conversion, then reads the next
amount.

	Large data DIFF and SORT are seldom run as in-memory operations --
they work line-by-line using files (in the case of some SORT algorithms,
many files: load 50-100 lines from source, sort in-memory, write to
file-1; repeat for file-2, -3, ... -n; when you have written to "n"
files, start back with the first file... Then do an -n file merge to
another n-files... Repeat until there is only one output file)
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

RE: Python does not take up available physical memory Pradipto Banerjee <pradipto.banerjee@adainvestments.com> - 2012-10-19 14:03 -0500
  Re: Python does not take up available physical memory Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2012-10-19 23:34 +0200
    Re: Python does not take up available physical memory Alain Ketterlin <alain@dpt-info.u-strasbg.fr> - 2012-10-20 07:09 +0200
  Re: Python does not take up available physical memory Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-19 22:12 +0000
    RE: Python does not take up available physical memory Pradipto Banerjee <pradipto.banerjee@adainvestments.com> - 2012-10-19 17:20 -0500
    Re: Python does not take up available physical memory Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-10-19 20:40 -0400
    RE: Python does not take up available physical memory Pradipto Banerjee <pradipto.banerjee@adainvestments.com> - 2012-10-21 09:14 -0500

csiph-web