Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Date: Fri, 27 Feb 2015 15:52:24 -0500
From: Dave Angel <davea@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Newbie question about text encoding
References: <aae131a7-29a1-4f79-ac16-d1e223616c51@googlegroups.com> <CAPTjJmoSZm8xRxeq-8G5KOKWddQxq23ieWqsY+jjCJXuY3DP0A@mail.gmail.com> <201502241620.t1OGKf4n002146@fido.openend.se> <54ECB134.5090304@davea.name> <201502241945.t1OJjshO013092@fido.openend.se> <201502241957.t1OJvrJS015604@fido.openend.se> <mailman.19148.1424810518.18130.python-list@python.org> <ef520397-b1f0-47bf-8d24-585a9ba230e2@googlegroups.com> <CAPTjJmreaPu7MZQgmbFNnhhg9R6w9dHPPo=yBbMoG85HxK+H_Q@mail.gmail.com> <mailman.19274.1424970167.18130.python-list@python.org> <00fbd940-52f6-44e2-bf08-b9f35c12e73f@googlegroups.com> <54efc2c8$0$12986$c3e8da3$5496439d@news.astraweb.com> <mailman.19299.1425002275.18130.python-list@python.org> <54f00787$0$12979$c3e8da3$5496439d@news.astraweb.com> <mailman.19302.1425022259.18130.python-list@python.org> <54f05aff$0$12980$c3e8da3$5496439d@news.astraweb.com> <54F078FD.5030801@davea.name> <mailman.19307.1425046945.18130.python-list@python.org> <mcq4a9$29g$1@speranza.aioe.org>
In-Reply-To: <mcq4a9$29g$1@speranza.aioe.org>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.19316.1425070352.18130.python-list@python.org>
Lines: 80
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:86590

On 02/27/2015 11:00 AM, alister wrote:
> On Sat, 28 Feb 2015 01:22:15 +1100, Chris Angelico wrote:
>
>>
>> If you're trying to use the pagefile/swapfile as if it's more memory ("I
>> have 256MB of memory, but 10GB of swap space, so that's 10GB of
>> memory!"), then yes, these performance considerations are huge. But
>> suppose you need to run a program that's larger than your available RAM.
>> On MS-DOS, sometimes you'd need to work with program overlays (a concept
>> borrowed from older systems, but ones that I never worked on, so I'm
>> going back no further than DOS here). You get a *massive* complexity hit
>> the instant you start using them, whether your program would have been
>> able to fit into memory on some systems or not. Just making it possible
>> to have only part of your code in memory places demands on your code
>> that you, the programmer, have to think about. With virtual memory,
>> though, you just write your code as if it's all in memory, and some of
>> it may, at some times, be on disk. Less code to debug = less time spent
>> debugging. The performance question is largely immaterial (you'll be
>> using the disk either way), but the savings on complexity are
>> tremendous. And then when you do find yourself running on a system with
>> enough RAM? No code changes needed, and full performance. That's where
>> virtual memory shines.
>> ChrisA
>
> I think there is a case for bringing back the overlay file, or at least
> loading larger programs in sections
> only loading the routines as they are required could speed up the start
> time of many large applications.
> examples libre office, I rarely need the mail merge function, the word
> count and may other features that could be added into the running
> application on demand rather than all at once.
>
> obviously with large memory & virtual mem there is no need to un-install
> them once loaded.
>

I can't say how Linux handles it (I'd like to know, but haven't needed 
to yet), but in Windows (NT, XP, etc), a DLL is not "loaded", but rather 
mapped.  And it's not copied into the swapfile, it's mapped directly 
from the DLL.  The mapping mode is "copy-on-write" which means that 
read=only portions are swapped directly from the DLL, on first usage, 
while read-write portions (eg. static/global variables, relocation 
modifications) are copied on first use to the swap file.  I presume 
EXE's are done the same way, but never had a need to know.

If that's the case on the architectures you're talking about, then the 
problem of slow loading is not triggered by the memory usage, but by 
lots of initialization code.  THAT's what should be deferred for 
seldom-used portions of code.

The main point of a working-set-tuner is to group sections of code 
together that are likely to be used together.  To take an extreme case, 
all the fatal exception handlers should be positioned adjacent to each 
other in linear memory, as it's unlikely that any of them will be 
needed, and the code takes up no time or space in physical memory.

Also (in Windows), a DLL can be pre-relocated, so that it has a 
preferred address to be loaded into memory.  If that memory is available 
when it gets loaded (actually mapped), then no relocation needs to 
happen, which saves time and swap space.

In the X86 architecture, most code is self-relocating, everything is 
relative.  But references to other DLL's and jump tables were absolute, 
so they needed to be relocated at load time, when final locations were 
nailed down.

Perhaps the authors of bloated applications have forgotten how to do 
these, as the defaults in the linker puts all DLL's in the same 
location, meaning all but the first will need relocating.  But system 
DLL's  are (were) each given unique addresses.

On one large project, I added the build step of assigning these base 
addresses.  Each DLL had to start on a 64k boundary, and I reserved some 
fractional extra space between them in case one would grow.  Then every 
few months, we double-checked that they didn't overlap, and if necessary 
adjusted the start addresses.  We didn't just automatically assign 
closest addresses, because frequently some of the DLL's would be updated 
independently of the others.
-- 
DaveA