Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #46365

Re: usage of os.posix_fadvise

Show all headers | View raw

Hi,

Wolfgang Maier <wolfgang.maier <at> biologie.uni-freiburg.de> writes:
> 
> Dear all,
> I was just experimenting for the first time with os.posix_fadvise(), which
> is new in Python3.3 . I'm reading from a really huge file (several GB) and I
> want to use the data only once, so I don't want OS-level page caching. I
> tried os.posix_fadvise with the os.POSIX_FADV_NOREUSE and with the
> os.POSIX_FADV_DONTNEED flags, but neither seemed to have any effect on the
> caching behaviour of Ubuntu (still uses all available memory to page cache
> my I/O).
> Specifically, I was trying this:
> 
> import os
> fd = os.open('myfile', os.O_RDONLY)
> # wasn't sure about the len parameter in fadvise,
> # so thought I just use it on the first 4GB
> os.posix_fadvise(fd, 0, 4000000000, os.POSIX_FADV_NOREUSE) # or DONTNEED

The Linux version of "man posix_fadvise" probably holds the answer:

"In kernels before 2.6.18, POSIX_FADV_NOREUSE had the same semantics
as POSIX_FADV_WILLNEED.  This was probably a bug; since kernel
2.6.18, this flag is a no-op."

"POSIX_FADV_DONTNEED attempts to free cached pages associated with the
specified region.  This is useful, for example, while streaming large
files.  A program may periodically request the kernel to free cached
data that has already been used, so that more useful cached pages  are
not discarded instead."

So, in summary:

- POSIX_FADV_NOREUSE doesn't do anything on (modern) Linux kernels
- POSIX_FADV_DONTNEED must be called *after* you are done with a range of
  data, not before you read it (note that I haven't tested to confirm it :-))

Regards

Antoine.

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: usage of os.posix_fadvise Antoine Pitrou <solipsis@pitrou.net> - 2013-05-29 10:51 +0000

csiph-web