Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.180 X-Spam-Level: * X-Spam-Evidence: '*H*': 0.64; '*S*': 0.00; 'cc:addr:python-list': 0.10; 'file,': 0.15; 'binary.': 0.16; 'inverse': 0.16; 'mmap': 0.16; 'simplest': 0.16; 'time"': 0.16; 'wrote:': 0.17; '>>>': 0.18; 'windows': 0.19; "i'd": 0.22; 'cc:2**0': 0.23; 'paul': 0.24; 'tried': 0.25; 'cc:addr:python.org': 0.25; 'header:In-Reply-To:1': 0.25; 'header:User-Agent:1': 0.26; 'am,': 0.27; 'record': 0.28; 'writes:': 0.29; 'file': 0.32; 'thanks': 0.34; 'really': 0.36; 'but': 0.36; 'subject: (': 0.36; 'possible': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'performance': 0.39; 'received:192': 0.39; 'little': 0.39; 'received:192.168': 0.40; 'subject:-': 0.40; 'think': 0.40; 'map': 0.61; 'first': 0.61; "you'll": 0.62; 'more': 0.63; 'tricks': 0.65; 'forward': 0.66; 'header:Reply-To:1': 0.68; 'wish': 0.70; 'received:74.208': 0.71; 'reply-to:no real name:2**0': 0.72; 'now...': 0.84; 'subject:Fast': 0.84; 'subject:read': 0.84; 'subject:write': 0.84; 'try.': 0.91 Date: Sun, 28 Oct 2012 07:18:01 -0400 From: Dave Angel User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120912 Thunderbird/15.0.1 MIME-Version: 1.0 To: Virgil Stokes Subject: Re: Fast forward-backward (write-read) References: <5086AA35.4000509@it.uu.se> <5086C156.40707@tim.thechases.com> <5086D13F.80006@it.uu.se> <5086DA4D.4060204@tim.thechases.com> <7xr4ooah0t.fsf@ruckus.brouhaha.com> <50879555.7000200@it.uu.se> In-Reply-To: <50879555.7000200@it.uu.se> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:k3nUv/4LCaPT/JfxRHp2d4tuGz2ymPK2mcKW6bzOVOR S04YVPV09CJCa/bgo7HuqWUnhBbx2aXUwuagAcVkX2Q+/k+L4X JWWMGjZOb1yQNhTrwBzcejnR4cyxaJP/e1cnP/PV2MbIstWJVj /dXMgsoqFYu6b3k+eZukYX+pq06CiYfIMTq+o9Lc/b0P3L7t2K LXz2aHSmnoO5hZ1H5qCYXSWkNC5OatVD5rJ/ZAQfgapVC7hyBE BKdoi1PDQ7eF9Gi7JLZhtYv3JjuAlUTyR0O7TT2bQz3EdLShzv bWLsBfsoCtE9uEOpeglev33bFnGA7lNilgYIyIJJM94w+6WQQ= = Cc: "python-list@python.org" X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: d@davea.name List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 21 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1351423115 news.xs4all.nl 6939 [2001:888:2000:d::a6]:34829 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:32310 On 10/24/2012 03:14 AM, Virgil Stokes wrote: > On 24-Oct-2012 01:46, Paul Rubin wrote: >> Virgil Stokes writes: >>> Yes, I do wish to inverse the order, but the "forward in time" file >>> will be in binary. >> I really think it will be simplest to just write the file in forward >> order, then use mmap to read it one record at a time. It might be >> possible to squeeze out a little more performance with reordering tricks >> but that's the first thing to try. > Thanks Paul, > I am working on this approach now... If you're using mmap to map the whole file, you'll need 64bit Windows to start with. I'd be interested to know if Windows will allow you to mmap 100gb at one stroke. Have you tried it, or are you starting by figuring how to access the data from the mmap? -- DaveA