Groups | Search | Server Info | Login | Register

Groups > perl.perl5.porters > #99832

Re: `cpan` unusable on small-RAM machines

Show all headers | View raw

Can you define what is considered "small-RAM machines"? 256MB? 512MB?

Although I do support 100% a modern, in core, "install me some modules 
from CPAN" script. If that's something you work on I might be interested 
in contributing.

-- Scottchiefbaker

On 1/29/2026 5:39 AM, Paul "LeoNerd" Evans wrote:
> On Tue, 14 Nov 2023 14:17:37 +0000
> "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk> wrote:
>
>> And on closer inspection, when it fails, it sits quite a while
>> spinning CPU and (presumably) burning memory. It get as far as
>> printing "Lockfile removed." nicely quickly, then waits aaaaages
>> (over 10 seconds) before the OOM-killer kills it.
>>
>>    All tests successful.
>>    Files=8, Tests=43,  3 wallclock secs ( 0.21 usr  0.02 sys +  2.94
>> cusr  0.34 csys =  3.51 CPU) Result: PASS
>>    Lockfile removed.
>>                        <== it waits for over 10 seconds here...
>>    Killed
> This issue came up again for me the other day, and I talked about it on
> #p5p. With the help of some interesting debugging ideas from alh,
> mauke, ilmari (and maybe some others whose names I have forgotten -
> sorry if I missed you), we think we may have narrowed down what is
> going on.
>
> Fairly early on in its startup, the main `cpan` process loads the
> entire metadata database into memory. It's somewhat big, but that's OK
> - it still fits into memory and doesn't upset things too much (as
> evidenced by the fact that the `cpan> look` command can run just fine,
> for example). Next, it `fork`s itself, before invoking some
> sub-process things via system(). This results in us having *two*
> processes (the original parent and a child), both with that large
> memory footprint. This is still fine - because of Linux copy-on-write.
>
> But now we have two processes, *both* of which have an END block which
> is going to do a lot of cleanup that touches a lot of memory, and we
> believe here is the time that it gets OOM killed. Each process has to
> suddenly grow its own copy of all those pages, because the SvREFCNT of
> all those SVs is going to be decremented to throw them away.
>
> Somewhat ironic that perl gets OOM killed during the process of it
> trying to throw away all its memory.
>
> Anyway, we don't have a firm idea of how to fix this yet, but we did
> poke at a few possible thoughts:
>
>    * Use POSIX::_exit instead of regular exit() from that inner process
>
>    * Use exec() instead of system() to invoke commands from that inner
>      process
>
>    * fork() a sidecar process early in startup before loading that
>      metadata database, and perform all the DB queries inside that via
>      some pipes, to ensure the master controller process doesn't have a
>      big memory footprint
>
> I may have a look into some of these on my RPi machines.
>
>
> However, I'm still not super inclined to actually continue maintaining
> CPAN.pm *itself*, because every time I try to dive into the code I find
> it a huge twisty nightmare of decades of accumulated cruft. It's hard
> to work out what it is supposed to be doing or where or why. I still
> feel that the time is well past due, to consider building a brand new
> cleanroom small/neat/efficient "install me some modules from CPAN"
> client to live in perl core. If I am going to put in a nontrivially
> small amount of effort, it is that which I'd prefer to work on.
>

Back to perl.perl5.porters | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Re: `cpan` unusable on small-RAM machines leonerd@leonerd.org.uk ("Paul \"LeoNerd\" Evans") - 2026-01-29 13:39 +0000
  Re: `cpan` unusable on small-RAM machines scott@perturb.org (Scott Baker) - 2026-01-29 08:16 -0800
  Re: `cpan` unusable on small-RAM machines philippe@bruhat.net ("Philippe Bruhat (BooK)") - 2026-01-30 10:14 +0100

csiph-web