Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > muc.lists.netbsd.tech.userlevel > #11824 > unrolled thread

Improving the performance of system boot by eliding unused rc.d scripts and caching the result

Back to article view | Back to muc.lists.netbsd.tech.userlevel

Contents

  Improving the performance of system boot by eliding unused rc.d scripts and caching the result Jason Thorpe <thorpej@me.com> - 2026-08-17 21:46 -0700
    Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Edgar Fuß <ef@math.uni-bonn.de> - 2026-08-18 11:07 +0200
      Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Jason Thorpe <thorpej@me.com> - 2026-08-18 06:07 -0700
    Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Greg Troxel <gdt@lexort.com> - 2026-08-18 06:41 -0400
      Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Jason Thorpe <thorpej@me.com> - 2026-08-18 06:44 -0700
        Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Mouse <mouse@Rodents-Montreal.ORG> - 2026-08-18 09:51 -0400
          Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Jason Thorpe <thorpej@me.com> - 2026-08-18 06:58 -0700
          Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result David Holland <dholland-tech@netbsd.org> - 2026-08-19 04:33 +0000
            barrier scripts (was: Improving the performance of system boot by eliding unused rc.d scripts and caching the result) Edgar Fuß <ef@math.uni-bonn.de> - 2026-08-19 11:20 +0200
            Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Greg Troxel <gdt@lexort.com> - 2026-08-19 06:43 -0400
              Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Jason Thorpe <thorpej@me.com> - 2026-08-19 06:16 -0700
              Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result David Holland <dholland-tech@netbsd.org> - 2026-08-20 16:03 +0000
                Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Greg Troxel <gdt@lexort.com> - 2026-08-20 12:35 -0400
                  Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Jason Thorpe <thorpej@me.com> - 2026-08-20 09:51 -0700
                  Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result David Holland <dholland-tech@netbsd.org> - 2026-08-20 21:09 +0000
            Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result "Simon J. Gerraty" <sjg@crufty.net> - 2026-08-19 14:07 -0700
        Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Greg Troxel <gdt@lexort.com> - 2026-08-19 07:07 -0400
          Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Jason Thorpe <thorpej@me.com> - 2026-08-19 08:47 -0700
            Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Robert Elz <kre@munnari.OZ.AU> - 2026-08-20 10:43 +0700
              Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Jason Thorpe <thorpej@me.com> - 2026-08-19 23:27 -0700
                Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Robert Elz <kre@munnari.OZ.AU> - 2026-08-21 04:37 +0700
              Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Edgar Fuß <ef@math.uni-bonn.de> - 2026-08-20 10:12 +0200
                Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Jason Thorpe <thorpej@me.com> - 2026-08-20 06:40 -0700
              Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Valery Ushakov <uwe@stderr.spb.ru> - 2026-08-20 12:38 +0300
                Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Robert Elz <kre@munnari.OZ.AU> - 2026-08-21 02:17 +0700
      Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Martin Neitzel <neitzel@hackett.marshlabs.gaertner.de> - 2026-08-21 01:38 +0200
    Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Robert Elz <kre@munnari.OZ.AU> - 2026-08-19 02:07 +0700
      "good" editors [was Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result] Mouse <mouse@Rodents-Montreal.ORG> - 2026-08-18 15:19 -0400
      Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Jason Thorpe <thorpej@me.com> - 2026-08-18 14:24 -0700
        Re: Improving the performance of system boot by eliding unused rc.d scripts and caching the result Robert Elz <kre@munnari.OZ.AU> - 2026-08-19 08:08 +0700
          shell quoting (was: Improving the performance of system boot by eliding unused rc.d scripts and caching the result) Edgar Fuß <ef@math.uni-bonn.de> - 2026-08-19 11:17 +0200
            Re: shell quoting (was: Improving the performance of system boot by eliding unused rc.d scripts and caching the result) David Holland <dholland-tech@netbsd.org> - 2026-08-20 16:16 +0000
              Re: shell quoting (was: Improving the performance of system boot by eliding unused rc.d scripts and caching the result) Ken Hornstein <kenh@pobox.com> - 2026-08-20 12:36 -0400
            Re: shell quoting (was: Improving the performance of system boot by eliding unused rc.d scripts and caching the result) Robert Elz <kre@munnari.OZ.AU> - 2026-08-21 12:37 +0700
              Re: shell quoting Jarle Greipsland <jarle.greipsland@norid.no> - 2026-09-09 08:36 +0200
                Re: shell quoting Edgar Fuß <ef@math.uni-bonn.de> - 2026-09-09 10:28 +0200
                  Re: shell quoting Robert Elz <kre@munnari.OZ.AU> - 2026-09-09 18:42 +0700
                    Re: shell quoting Edgar Fuß <ef@math.uni-bonn.de> - 2026-09-09 14:42 +0200
                      Re: shell quoting kre@munnari.OZ.AU - 2026-09-09 20:03 +0700
                      Re: shell quoting Robert Elz <kre@munnari.OZ.AU> - 2026-09-09 20:21 +0700
                Re: shell quoting Robert Elz <kre@munnari.OZ.AU> - 2026-09-09 18:36 +0700
                  Re: shell quoting Edgar Fuß <ef@math.uni-bonn.de> - 2026-09-09 17:28 +0200
                    Re: shell quoting Edgar Fuß <ef@math.uni-bonn.de> - 2026-09-09 17:31 +0200
                    Re: shell quoting Robert Elz <kre@munnari.OZ.AU> - 2026-09-10 08:12 +0700
                      Re: shell quoting Mouse <mouse@Rodents-Montreal.ORG> - 2026-09-09 21:38 -0400
                        Re: shell quoting Robert Elz <kre@munnari.OZ.AU> - 2026-09-10 10:10 +0700
                  Re: shell quoting Jarle Greipsland <jarle.greipsland@norid.no> - 2026-09-09 18:56 +0200
                    Re: shell quoting Robert Elz <kre@munnari.OZ.AU> - 2026-09-10 08:51 +0700

Page 1 of 3  [1] 2 3  Next page →

#11824 — Improving the performance of system boot by eliding unused rc.d scripts and caching the result

I was chatting with the home-brewer who built the wrap030 and he noted that he was experiencing excruciatingly long boot times, roughly 14 minutes from power on to login prompt.  The wrap030 is a 25MHz 68030 system with a relatively constrained I/O subsystem: 8 16550 UARTs and a PIO-only ATA disk interface.  The current incarnation of the machine has 16MB of DRAM.

He took some notes about what was taking a long time and a few things stood out: ldconfig and motd.

ldconfig was addressed separately, but motd?  That script doesn’t do much!

Well, as it turns out, what he was really observing with motd was “there are lots of things run around the same time as motd that aren’t being used”.  That is, their rc.d scripts were being run only for them no decide to not do any work because their service was not enabled.  Some of the work they decided not to do was “report progress”, so it just looked like motd was taking a really long time.

After instrumenting the rc.d script incovations, we discovered that the scripts that didn’t actually do anything were taking quite a bit of time as far as no-ops go… anywhere between 3 to 6 seconds each, sometimes more.  Some of this was going into spawning sub-shells, some of it was going into reading the script, some of it was going into determing if the service was already running (which rc.subr does always for every rc.d script).  All on a machine that is very I/O constrained.  It was death by 1000 not-exactly-paper-cuts.

I stewed on the problem for a little while and came up with a solution: Compute the set of scripts that will do actual work, cache that result, and then use that cached ordered list to run *only those* scripts at boot time.  A quick proof-of-concept was thrown together and initial tests looked promising, so I worked on this proposed solution.  Initial tests of this solution on the wrap030 were quite shocking.  After the cached script was was computed, boot time dropped by nearly 50%, from ~13 minutes to ~7 minutes (!!).  I decided to give it a whirl on my AlphaStation 200 4/233.  It has faster I/O than the wrap030, and has a few heavy-hitters in the boot process (ntpd, sshd, IPv6 DAD, fontconfig cache, etc.) that take up quite a bit of time on their own, but even on that machine, “date-to-date” boot time dropped from 2min 20sec to 2min flat.

These changes are essentially confined to just 2 files: /etc/rc.subr, which implements the meat of the “rcorder.cache” as a set of shell functions plus a default method to determine “this script does useful work”, and some small changes to /etc/rc to use the cache and update it, if appropriate.  The cache is considered out-of-date if any of /etc/rc.d, /etc/rc.conf, /etc/defaults/rc.conf, or /etc/rc.conf.d/* are newer than the “rcorder.cache”, which is stored at /etc/rcorder.cache.  Yes, I know, but the cache file has to be in /etc because it’s the only location guaranteed to be available.  It has provisions for not throwing useless errors if a read-only root file system is employed, and has an /etc/rc.conf knob to disable it.  There is an /etc/rc.d/rcorder_cache that does nothing at boot time, but can be used to query the in-use status of the cache (“status”) or rebuild the cache explicitly (“reload”).

“But Jason, no one is complaining about boot time in retro-emulators!” I can hear some of you saying.  And, for the most part you’re right.  Emulators mask this problem in two ways: they are not typically cycle accurate (on a real machine, a memory read cycle from a DRAM location takes less than half the time of a memory read cycle from the ATA data port), and the disk I/O on an emulated machine is typically an “instantaenous” affair.  These problems are best observed on actual machines, and I suspect 32-bit SPARC, VAX, practically all supported m68k systems, and othwise I/O constrained machines like Shark (or anything that has NFS root) to benefit most from these changes.

The issue is captured here: https://gnats.netbsd.org/60607

The proposed changes, here: https://www.netbsd.org/~thorpej/rcorder-cache-diff-v2.txt

Discutez-en, svp!

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [next] | [standalone]

#11825

> Some of this was going into spawning sub-shells
Did you try rc_fast_and_loose?

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11827

> On Aug 18, 2026, at 2:07 AM, Edgar Fuß <ef@math.uni-bonn.de> wrote:
> 
>> Some of this was going into spawning sub-shells
> Did you try rc_fast_and_loose?

Yes, I even mentioned it in the bug report I filed to track the issue.  It saved some time (2 minutes out of 13), but did not have nearly the same impact because rc.d scripts do more than just cause sub-shells to be spawned.  Given that the setting has this warning:

#       NOTE:   USE THIS AT YOUR OWN RISK; A ROGUE COMMAND
#               MAY INADVERTENTLY PREVENT BOOT TO MULTIUSER.

it just doesn’t seem like the best solution to the problem.  (Slower *and* dangerous?  Maybe when I was younger, but certainly not now.)

It’s easy to see why rc_fast_and_loose isn’t that much faster:

icarus# rcorder -s nostart /etc/rc.d/* | wc -l
     128
icarus# wc -l /etc/rcorder.cache
      45 /etc/rcorder.cache
icarus# 

Not even visiting the files that do no useful work saves a lot more CPU cycles and I/O than throwing caution to the wind and slurping them all into a single process.

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11826

Can you explain the essence of how the code evaluates "script does
useful work"?

The obvious way to put it is "is the effect on the system (other than
cpu time, filesystem reads, effects on caches) from running this script
the same as the effect of not running it"?

If there is a rule that each script must have a yes/no variable
controlling actions, then I can see this optimization.  Or really,
skipping any script that does have a variable.  But, rc.d scripts sort
of seem like arbitrary shell commands.  I therefore wonder if we have
rules for rc.d scripts that enable this kind of optimization, or if some
subset of optimizable rc.d scripts are recognized.

How does this interact with rc.d scripts from pkgsrc or elsewhere, that
do not necessarily follow rules?

(Also, it seems obvious that if you commit this, it should default to
off at first, except perhaps for particularly slow arches.)

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11828

> On Aug 18, 2026, at 3:41 AM, Greg Troxel <gdt@lexort.com> wrote:
> 
> Can you explain the essence of how the code evaluates "script does
> useful work"?

There’s a comment in the code that describies it:

+#	doeswork	Returns 0 if the script does work that's needed
+#			for boot, non-zero otherwise.  Scripts are considered
+#			to do work if either their rcvar is set to YES or
+#			if they do not have a defined rcvar.
+#

So let me explain the reasoning.  If a script defines a controlling rcvar, then that script, by definition, has been requested to do nothing if the rcvar evaluates to NO.  Scripts that do not define an rcvar fall into three categories:

- scripts that always do some sort of work (e.g. mountcritlocal)
- scripts that make some other determination as to whether or not they should do work (e.g. ccd)
- the barrier scripts (e.g. LOGIN)

The barrier scripts are needed only for ordering, and the dismissal of non-useful scripts occurs after ordering, but in my implementation they remain in the rcorder.cache bcause:

(a) there’s not really a fast or simple way to distinguish them from the second category, and

(2) given the number of truly non-useful scripts that typically get elided, having a couple in there that serve to document the sequencing points seems like a good trade-off.

> The obvious way to put it is "is the effect on the system (other than
> cpu time, filesystem reads, effects on caches) from running this script
> the same as the effect of not running it"?

I guess I can really distill it down to: “A script is considered to do no useful work only if it definitively tells us so.”  And it does so by self-reporting that its rcvar is set to NO.

> If there is a rule that each script must have a yes/no variable
> controlling actions, then I can see this optimization.  Or really,
> skipping any script that does have a variable.  But, rc.d scripts sort
> of seem like arbitrary shell commands.  I therefore wonder if we have
> rules for rc.d scripts that enable this kind of optimization, or if some
> subset of optimizable rc.d scripts are recognized.

The basic rule for rc.d scripts that work in our system is “use rc.subr”.  Any script that does will get a safe default for “does useful work”.  Any script that doesn’t probably doesn’t actually work properly as it is today.  A main design feture of our rc.d system is that scripts that don’t provide an explcit action for one of the directives get a widely-cast net of reasonable default behavior (and yes, I went back and read Luke’s USENIX paper again to provide maximum insurance against violating any religious tenents while working on this problem).

> How does this interact with rc.d scripts from pkgsrc or elsewhere, that
> do not necessarily follow rules?

Well, the most important program in pkgsrc that supplies rc.d scripts (sysutils/nabud, of course) definitely follows the rules.  A cursory audit of main NetBSD workhorse machine has a few pkgsrc-installed rc.d scripts, and they all DTRT as well.

The one situation where it could fall over is “some random rc.d script doesn’t use rc.subr at all”, and this it will not respond to the “doeswork” directive.  But even if I invert the sense to a “canskip” directive, some random rc.d script that doesn’t use rc.subr at all could choose to play Towers of Hanoi rather than exit with an error status.

I guess my point is the only rule a script has to follow is “use rc.subr”, which seems to be an *incredibly* low bar (because if they don’t, there’s already a myriad of ways those scripts could fall over).  If it follows that one rule, then the only way it gets optimized out is if is uses a control variable and that control variable contains the value that explcitiy says “yo script, you are to do no work”.

> (Also, it seems obvious that if you commit this, it should default to
> off at first, except perhaps for particularly slow arches.)

Of course, this has the side-effect of reducing the amount of dog food-induced problem finding and is yet another step that people have to do in order to make their systems run fast, but ok, sure.

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11829

> - the barrier scripts (e.g. LOGIN)

> The barrier scripts are needed only for ordering, [...]

If this really bothers someone, perhaps there could be a way for a
script to declare itself a barrier script?  Perhaps something a la
make's .PHONY, perhaps a naming convention, perhaps something else.

Or perhaps the ordering could be enforced some other way?  (I haven't
looked at rcorder in a quite a while, so I could be spouting nonsense
here.)

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse@rodents-montreal.org
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11830

> On Aug 18, 2026, at 6:51 AM, Mouse <mouse@Rodents-Montreal.ORG> wrote:
> 
>> - the barrier scripts (e.g. LOGIN)
> 
>> The barrier scripts are needed only for ordering, [...]
> 
> If this really bothers someone, perhaps there could be a way for a
> script to declare itself a barrier script?  Perhaps something a la
> make's .PHONY, perhaps a naming convention, perhaps something else.

The number of barriers is tiny, and it doesn’t seem like the right engineering trade-off to add more decoration support for them in oder to be able to safely elide them from the cached result.

My goal here was to make a small, safe, non-invasive change, and I explcitly wanted to avoid any new functionality for rc.d scripts to adopt[*].

[*] Ok, to be fair, because a new directive keyword is added (“doeswork”), scripts technically can include a doeswork_cmd variable that overrides the default behavior.  But this is merely an intentional side-effect of how rc.subr works.

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11838

On Tue, Aug 18, 2026 at 09:51:44AM -0400, Mouse wrote:
 > > - the barrier scripts (e.g. LOGIN)
 > 
 > > The barrier scripts are needed only for ordering, [...]
 > 
 > If this really bothers someone, perhaps there could be a way for a
 > script to declare itself a barrier script?  Perhaps something a la
 > make's .PHONY, perhaps a naming convention, perhaps something else.

Is there any reason for the barrier scripts to physically exist?

I remember rewriting rcorder at mrg's suggestion ages ago, and then
never merging it, but I've completely forgotten both what it was about
and why it didn't get merged... but there's definitely no need to have
an empty/vacuous script just to have an ordering hook.

-- 
David A. Holland
dholland@netbsd.org

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11840 — barrier scripts (was: Improving the performance of system boot by eliding unused rc.d scripts and caching the result)

> Is there any reason for the barrier scripts to physically exist?
Well, first rcorder needs them fot the PROVIDE:/REQUIRE: lines.
Second, there's a comment in them explaining what the barrier is.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11841

David Holland <dholland-tech@netbsd.org> writes:

> On Tue, Aug 18, 2026 at 09:51:44AM -0400, Mouse wrote:
>  > > - the barrier scripts (e.g. LOGIN)
>  > 
>  > > The barrier scripts are needed only for ordering, [...]
>  > 
>  > If this really bothers someone, perhaps there could be a way for a
>  > script to declare itself a barrier script?  Perhaps something a la
>  > make's .PHONY, perhaps a naming convention, perhaps something else.
>
> Is there any reason for the barrier scripts to physically exist?

Picking one arbitrarily,

  # $NetBSD: LOGIN,v 1.8 2022/03/02 01:55:18 gutteridge Exp $

  # PROVIDE: LOGIN
  # REQUIRE: DAEMON

I can see how you can impute that LOGIN provides LOGIN, but someplace
the information the LOGIN requires DAEMON has to exist.

How would you store and configuration manage the ordering requirements
among barrier scripts?

This is 6 tiny files.  I don't think it's a problem, and it's easier for
humans to read and reason about rc.d scripts with them present, instead
of imputed information.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11843

> On Aug 19, 2026, at 3:43 AM, Greg Troxel <gdt@lexort.com> wrote:
> 
> This is 6 tiny files.  I don't think it's a problem, and it's easier for
> humans to read and reason about rc.d scripts with them present, instead
> of imputed information.

Yes, this.  And their presence doesn’t really present a performance problem as kre@ noted (because they don’t slurp in any actual code).

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11853

On Wed, Aug 19, 2026 at 06:43:20AM -0400, Greg Troxel wrote:
 > > Is there any reason for the barrier scripts to physically exist?
 > 
 > Picking one arbitrarily,
 > 
 >   # $NetBSD: LOGIN,v 1.8 2022/03/02 01:55:18 gutteridge Exp $
 > 
 >   # PROVIDE: LOGIN
 >   # REQUIRE: DAEMON
 > 
 > I can see how you can impute that LOGIN provides LOGIN, but someplace
 > the information the LOGIN requires DAEMON has to exist.
 > 
 > How would you store and configuration manage the ordering requirements
 > among barrier scripts?

Just off the top of my head, you could:

 - hardwire the barrier sequence (admittedly not a great plan)
 - put it on the rcorder command line
 - put it all in a single non-executable file

-- 
David A. Holland
dholland@netbsd.org

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11855

David Holland <dholland-tech@netbsd.org> writes:

> On Wed, Aug 19, 2026 at 06:43:20AM -0400, Greg Troxel wrote:
>  > > Is there any reason for the barrier scripts to physically exist?
>  > 
>  > Picking one arbitrarily,
>  > 
>  >   # $NetBSD: LOGIN,v 1.8 2022/03/02 01:55:18 gutteridge Exp $
>  > 
>  >   # PROVIDE: LOGIN
>  >   # REQUIRE: DAEMON
>  > 
>  > I can see how you can impute that LOGIN provides LOGIN, but someplace
>  > the information the LOGIN requires DAEMON has to exist.
>  > 
>  > How would you store and configuration manage the ordering requirements
>  > among barrier scripts?
>
> Just off the top of my head, you could:
>
>  - hardwire the barrier sequence (admittedly not a great plan)
>  - put it on the rcorder command line
>  - put it all in a single non-executable file

I can see those working -- but I don't see how that's an improvement,
weighing the benefit of dropping these tiny files and invoking them vs
the cost of adding additional mechanisms, absent benchmarks that show
the current situation hurts.

After all, anyone truly concerned about performance would port systemd
to NetBSD and vibe-code optimizations.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11857

> On Aug 20, 2026, at 9:35 AM, Greg Troxel <gdt@lexort.com> wrote:
> 
> I can see those working -- but I don't see how that's an improvement,
> weighing the benefit of dropping these tiny files and invoking them vs
> the cost of adding additional mechanisms, absent benchmarks that show
> the current situation hurts.

Right, to be clear, I don’t view this barrier files as a problem in the slightest.  I’m almost sorry I mentioned them.

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11859

On Thu, Aug 20, 2026 at 12:35:42PM -0400, Greg Troxel wrote:
 > >  > How would you store and configuration manage the ordering requirements
 > >  > among barrier scripts?
 > >
 > > Just off the top of my head, you could:
 > >
 > >  - hardwire the barrier sequence (admittedly not a great plan)
 > >  - put it on the rcorder command line
 > >  - put it all in a single non-executable file
 > 
 > I can see those working -- but I don't see how that's an improvement,
 > weighing the benefit of dropping these tiny files and invoking them vs
 > the cost of adding additional mechanisms, absent benchmarks that show
 > the current situation hurts.

Running a shell on the empty file is not free, even if it's not
expensive. (AIUI, without the fast-and-loose setting, each one of
these is run in at least a subshell if not a fresh exec.)

Conversely, rcorder(8) is compiled code and a bit of extra logic there
is by comparison free... plus there you're saving opening and reading
a few files so you're still ahead.

 > After all, anyone truly concerned about performance would port systemd
 > to NetBSD and vibe-code optimizations.

Right, obviously :-)

-- 
David A. Holland
dholland@netbsd.org

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11847

On Wed, 19 Aug 2026 04:33:58 +0000, David Holland writes:
>On Tue, Aug 18, 2026 at 09:51:44AM -0400, Mouse wrote:
> > > - the barrier scripts (e.g. LOGIN)
> > 
> > > The barrier scripts are needed only for ordering, [...]
> > 
> > If this really bothers someone, perhaps there could be a way for a
> > script to declare itself a barrier script?  Perhaps something a la
> > make's .PHONY, perhaps a naming convention, perhaps something else.
>
>Is there any reason for the barrier scripts to physically exist?

Can they do their job otherwise?

A possibly extreme example (perhaps only relevant as FreeBSD borrowed
rc.subr et al from NetBSD), in Junos we have to run rcorder a few times
during boot.  The software is provided via packages which contain
filesystem images which must be verified and mounted.  An /etc/rc.d/*
script does not exist until its package has been mounted.

When the system first boots the kernel's rootfs mounts only a core
runtime and libs packages, so only a small subset of /etc/rc.d/* can be
considered when rcorder is first run.
The barrier EARLY appears after the rc.d script which mounts the bulk of
the core packages, so rcorder needs to be recomputed at that point.
There are two other barriers LATE and LATER which also cause rcorder to
need to be recomputed.

Anyway; I favor leaving the barriers alone ;-)


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11842

Jason Thorpe <thorpej@me.com> writes:

> There’s a comment in the code that describies it:
>
> +#	doeswork	Returns 0 if the script does work that's needed
> +#			for boot, non-zero otherwise.  Scripts are considered
> +#			to do work if either their rcvar is set to YES or
> +#			if they do not have a defined rcvar.
> +#
>
> So let me explain the reasoning.  If a script defines a controlling rcvar, then that script, by definition, has been requested to do nothing if the rcvar evaluates to NO.  Scripts that do not define an rcvar fall into three categories:
>
> - scripts that always do some sort of work (e.g. mountcritlocal)
> - scripts that make some other determination as to whether or not they should do work (e.g. ccd)
> - the barrier scripts (e.g. LOGIN)

> I guess I can really distill it down to: “A script is considered to do no useful work only if it definitively tells us so.”  And it does so by self-reporting that its rcvar is set to NO.

That seems sound.

> The basic rule for rc.d scripts that work in our system is “use
> rc.subr”.  Any script that does will get a safe default for “does
> useful work”.  Any script that doesn’t probably doesn’t actually work
> properly as it is today.  A main design feture of our rc.d system is
> that scripts that don’t provide an explcit action for one of the
> directives get a widely-cast net of reasonable default behavior (and
> yes, I went back and read Luke’s USENIX paper again to provide maximum
> insurance against violating any religious tenents while working on
> this problem).

That is a rule in NetBSD but I do not expect it is followed by all
scripts installed by pkgsrc.

A different view is that the basic rule is that an rc.d script must
implement start, stop, status, reload, and should check a variable.
rc.subr is certainly a good library to use, and a convention, but it
isn't strictly necessary.


> The one situation where it could fall over is “some random rc.d script
> doesn’t use rc.subr at all”, and this it will not respond to the
> “doeswork” directive.  But even if I invert the sense to a “canskip”
> directive, some random rc.d script that doesn’t use rc.subr at all
> could choose to play Towers of Hanoi rather than exit with an error
> status.

Sure, but we can say that "invoked with a command that isn't understood"
should lead to quick error exit as an implied specification, far more
strongly than we can say that using rc.subr is an implied specification.

There are a lot of packages with scripts and many of them are old.

> I guess my point is the only rule a script has to follow is “use
> rc.subr”, which seems to be an *incredibly* low bar (because if they
> don’t, there’s already a myriad of ways those scripts could fall
> over).  If it follows that one rule, then the only way it gets
> optimized out is if is uses a control variable and that control
> variable contains the value that explcitiy says “yo script, you are to
> do no work”.

I don't see any downside to inverting the test.

>> (Also, it seems obvious that if you commit this, it should default to
>> off at first, except perhaps for particularly slow arches.)
>
> Of course, this has the side-effect of reducing the amount of dog
> food-induced problem finding and is yet another step that people have
> to do in order to make their systems run fast, but ok, sure.

I said "at first".   While every change that's committed is believed to
not cause regressions, it seems best to let the first 20 people opt in
and after it's been a month or so and there are no bugs on the table, it
seems ok to make it default.

Basically I don't think it's ok to push testing onto current-users,
until there's been enough testing that finding a bug would be very
surprising.  In this case, testing is all about environments you haven't
contemplated.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11844

> On Aug 19, 2026, at 4:07 AM, Greg Troxel <gdt@lexort.com> wrote:
> 
>> I guess I can really distill it down to: “A script is considered to do no useful work only if it definitively tells us so.”  And it does so by self-reporting that its rcvar is set to NO.
> 
> That seems sound.

Ok, it was a little more invasive, but I’ve made this more robust in the face of random scripts that care not for rules.  Skip-ability is now defined as “affirmatively outputs YES on stdout in response to the skipstart directive”; this required additional handling for scripts that declare themselves to be “interactive”.

So now a non-response or an error or Towers of Hanoi or the output of “fortune zippy” or whatever else you can or cannot think of will result in including the script in the boot process.

I’ve also incorporated kre’s feedback about cache-staleness (some additional delay observed, but overall it’s still a big win) and the various Shell programming best-practices he noted.

    https://www.netbsd.org/~thorpej/rcorder-cache-diff-v3.txt

I’ll see about dragging out my NeXT or Shark this evening for a diskless boot check.

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11848

    Date:        Wed, 19 Aug 2026 08:47:51 -0700
    From:        Jason Thorpe <thorpej@me.com>
    Message-ID:  <28EA56EB-75D2-4FE9-8FC4-D1959F1D61D9@me.com>

  |     https://www.netbsd.org/~thorpej/rcorder-cache-diff-v3.txt

Just a couple of minor comments about this version, and only (at least
before I start on it in detail) one actual change suggested.

+	if [ $_rc_update_rcorder_cache = YES ]; then

That expansion isn't quoted -- but that one is (should be) OK, as the
var is always set to either YES or NO, so unless IFS happened to acquire
one (or more) of 'Y' 'E' 'S' 'N' or 'O' in its value (very unlikely as
IFS can't be imported, it has to be set by the script if it is to be
changed) that one is safe enough.

However, as mentioned again below, quoting expansions, even when
not required, is generally the better thing to do.

 		if ! checkyesno ${rcvar}; then

That one is clearly there already, not part of your changes, but
that should be quoted, rcvar comes from the script, and could be
set to anything (to work it must be a 'name' (as defined by sh), but
if some package were not interested in it working, anything is possible).

+		skipstart)
+			if [ -n "$rcvar" ] && ! checkyesno ${rcvar}; then

And then the same there.  For the first reference, the (already there)
quoting is essential, in case rcvar is not set, & while the test ensures
it has a value before the 2nd reference, it still might be anything.

+				command echo YES

There I won't complain about the use of echo (though printf always
is generally better, just for the habit using it produces) as any
version of echo which doesn't properly output a simple alphabetic word
would be too broken to exist.  (Same for the NO case just after).

 	    && _has_rcorder_keyword interactive $_file

Another expansion that is there already (ie: not your change),
but should be quoted.

+	if _rc_order_use_cache && [ -f /etc/rcorder.cache ]; then
+		# Listed here in order of most-likely-to-change.
+		allf="/etc/rc.conf /etc/rc.conf.d /etc/rc.conf.d/"*
+		for d in ${rc_directories:-/etc/rc.d}; do
+			allf="$allf ${d}/"*

Those two var assignments to allf work just fine as written
but they would work just the same way, and look a little less
peculiar, if written:

+		allf="/etc/rc.conf /etc/rc.conf.d /etc/rc.conf.d/*"

and

+			allf="$allf ${d}/*"

The quotes in these (or something similar) are needed to avoid the
arrignment word ending at the white space, but that's all they achieve here.

No field splitting or filename expansions happen in assignments, so
the quotes are not protecting against that, nor does leaving the '*'
unquoted cause filename expansion to happen.   The two forms are
entirely equivalent (hence no change required).

That last one could even be written

			allf=$allf\ ${d}/*

with exactly the same effect, the only char in the line affected
by the quoting is the space (the " could not be changed to ' though
as then the $allf and $d would not be expanded, those need to be
either unquoted, or "" quoted to work).  On the other hand, the first
of these 2, which has no expansions, just words, could use '' quoting
(which is marginally more efficient inside sh).

[Aside: writing $allf (no braces) and ${d} (with braces) in the
    same command like that, when there is no reason to include the
    braces in either case (there is never a reason, except perhaps
    line length, to exclude them) is a kind of confusing style.
]

The quotes are stripped (as always) as just about the last operation
before a command is run (not quite last, as redirections, etc, happen
even later, but that's not relevant here) so what is assigned to allf
by the second of those is just the value of allf followed by a space,
then the value of d with /* appended to it.   No quotes.

+		for f in ${allf}; do

That's where the * gets expanded, ${allf} isn't quoted there, and
so undergoes field splitting, then each resulting field that contains
a shell meta character ('*' '?' or '[') gets subjected to filename expansion.

This is when the $d/* from above gets turned into a list of filenames.

Whether the '*' was quoted, or not, when it was assigned to allf, makes
no difference to anything, it is unquoted here, so gets expanded (if
there are any matches).   The only way to prevent that would be if the
character before the '*' in the value of allf was a '\'  (which we know
it isn't, as it is, when assigned, a '/' instead).

So, no change needed there, it all works as intended, just looks a bit odd,
perhaps gives a misleading impression of what is happening.

+	allf=$(for d in ${rc_directories:-/etc/rc.d}; do
+		test -d "$d" && command echo "${d}"/*;
+	done)

In contract to the above, to work as planned, that * does need to
be unquoted (the '/' does not, but that makes no difference to anything),
that is a word which is an arg for a command, and is subjected to
everything, no field splitting will happen, as the only expansion
is properly quoted (this is the "${d}"/* word I am discussing)
but the resulting (single) field contains a meta char (unquoted),
so filename expansion happens, so the result of that loop is a list
of filenames - in this case separated by newlines.

And even though that works just as planned, this is my one actual
suggestion for a change, in two ways, for different reasons.

First, as the script doesn't control ${rc_directories} it doesn't
control the values of $d so this is a place where printf should be
used rather than echo.

so, we could replace the middle line of that with

+		test -d "$d" && command printf '%s ' "${d}"/*;

[Aside: the ';' is not useful, and could be deleted, this is not C,
  expressions don't need to end in a ';' to turn them into commands,
  a newline works just as well as the terminator - but that's harmless.
  The ';' would be needed if the "done" were moved up to the same line -
  just the same as the ';' in the "for d ..." line, if the "do" were on the
  next line, that ';' wouldn't be needed - and none is needed, or
  even allowed, immediately after the "do" - that would make an empty
  command, and sh syntax in general does not allow those. 
]

One change there, is that now the list of filenames is separated
by spaces, rather than newlines.  That makes no difference to
anything, as both space and newline are in the normal value of IFS,
so both work the same way when we get to do field splitting in
the following line.

+	orderedf=$(rcorder -s nostart ${rc_rcorder_flags} ${allf})

This is one case where quotes are not wanted anywhere.   They're
not wanted for ${rc_rcorder_flags} for 2 reasons - first if there
are no flags, that arg needs to vanish completely, whereas quoting
it would leave a "" string instead, not what is desired; and second
to allow ${rc_rcorder_flags} to expand to multiple words if needed.

And for ${allf} we have a list of filenames we need to pass to
rcorder to consider, that list needs to be field split.   As
currently written in -v3 we really don't want filename expansion
any more though, as that has already happened.  If one of the rc.d
files happened to be named 'foo*' we want to rcorder 'foo*' not all
the files that have names starting with 'foo'.

There are two ways to prevent that, one would be to make the cmdsub
for the assignment to orderdf be

+	orderedf=$(set -f; rcorder -s nostart ${rc_rcorder_flags} ${allf})

which in some ways would be safest, as it would also prevent filename
expansion of the results of field splitting ${rc_rcorder_flags} - but
as that is a very low risk, I'd suggest leaving filename expansion
enabled there (+f - the default) and instead skipping the filename
expansion in the construction of the value of allf, by changing the
line above to:

+		test -d "$d" && command printf '%s ' "${d}/*"

so that allf ends up being just "rc.d/* " in the usual case
(rc_directories not set).  That is then field split with the
expansion of ${allf} (unquoted) in the command substitution,
(which in this case simply deletes the space, and leaves the
single field rc.d/* which then gets filename expanded (just
once, rather than twice, as the current -v3 version is doing).

Which of those 2 solutions gets adopted makes not a lot of difference,
but either the "set -f" version, or the quoted '*' in the earlier
assignment needs to happen for general safety.   I'd do the quoted
'*' version, as the risk from filename expansion of ${rc_rcorder_flags}
is minimal, and it causes the size of what is assigned to allf to be
much smaller (and that happens at in a higher level shell, the command
substitution still needs to expand it, but once done, that forked
shell simply vanishes, with its much longer string allocation along
with it, and the way sh works, no malloc() ever happens just for it,
saving just a tiny bit more).

+	if ! _rc_order_use_cache; then
+		_rc_ordered_script_list="$orderedf"

Another place where the quotes are harmless, and meaningless.

+		if [ x"$canskip" != xYES ]; then

The 'x' chars there are an ancient workaround, and no longer
serve any practical purpose, as long as one sticks to the defined
usages of test (which this is).   Once upon a time, those served
a purpose, now they are just annoying to read (still works the same
of course - but with longer strings to compare!)

+	if (command printf "" > /etc/rcorder.cache.tmp) 2>/dev/null; then

Here you don't need the 'command printf ""' at all, this is just
testing whether the file can be opened for writing, and creating it,
though that isn't essential, and for that 

+	if (> /etc/rcorder.cache.tmp) 2>/dev/null; then

would work just fine.   Unfortunately I think the () (subshell) is
needed, as redirection errors in scripts tend to cause the shell to
exit ... the subshell exiting is harmless, it had no more work to
do anyway, but the shell running the script can't be allowed to
just exit there.

+			command printf "${f}\n" >> /etc/rcorder.cache.tmp

Oh, I didn't see this one earlier, this is another that should be
fixed, (so I am actyally suggesting 2 changes that should be made, not
just the one (above) I thought I'd be making).

Used that way, printf is no better than echo would be.

Make that one:

+			command printf '%s\n' "${f}" >> /etc/rcorder.cache.tmp

so that "${f}" is just a string, no embedded formatting conversions applied.
(this is the same as in C, "printf(fmt)" with a 'char *fmt' string that comes
from who knows where).   The \n needs to be in the format in this case
to generate a newline, rather than the 2 chars '\' 'n' which it would be
if written as:

+			command printf %s "${f}\n" >> /etc/rcorder.cache.tmp

Though:

+			command printf %s "${f}"$'\n' >> /etc/rcorder.cache.tmp

would work if, for some obscure reason I can't imagine, that was
considered better (puts a newline character at the end of the string
handed to printf %s (which then is not adding one).


Then in the new rd.d script:

+name="rcorder_cache"
					(quoting 100% useless there)
+rcvar=$name

[....]

+rcorder_cache_status()
+{
+	if checkyesno ${rcvar}; then

In that one, as this scipt has total control over the value of
rcvar, it is acceptable to not quote it.   But it would be better
if it were, again, more for getting into the habit of always quoting
expansions, unless able to justify why quoting one wouldn't work.

+		if [ -f /etc/rcorder.cache ]; then
+			echo "/etc/rcorder.cache is in use."

For that, while (rc.subr's) echo() is fine, '' rather than "" would
be better (minimally better, but better) - less work is needed to
be done by sh with '' quoted strings than for "" ones (no looking
for \ or $ inside for example).   This is a case where the quoting
is not really needed at all, echo (and its printf based implementation)
handle multiple args just fine - but quoting it is slightly better, just
one arg to deal withm rather than 4 (in that case), and for the quoting
'' is better than "" when no embedded expansions are needed.

This is another case where sh is not C, in C '' makes ints, "" makes
arrays of char, in sh, everything (except operators) is a string
(internally an array of char) whether quoted or not, and if quoted
what kind of quoting was used merely affects what expansions can happen
inside the string.

There is no need to change this (or the other two in this function)
they will work as written with no issues, but as the aim of all of
this set of changes is to be as fast as possible, using '' over ""
might save a few microseconds on a slow processor (probably < 1 of
those on a fast one).

+load_rc_config $name

One more example where (becase the script owns the value of name)
quoting the expansion is not essential, but would be better, just
on general principle.

kre

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#11849

> On Aug 19, 2026, at 8:43 PM, Robert Elz <kre@munnari.OZ.AU> wrote:
> 
>    Date:        Wed, 19 Aug 2026 08:47:51 -0700
>    From:        Jason Thorpe <thorpej@me.com>
>    Message-ID:  <28EA56EB-75D2-4FE9-8FC4-D1959F1D61D9@me.com>
> 
>  |     https://www.netbsd.org/~thorpej/rcorder-cache-diff-v3.txt

<snip>

Feedback incorporated:

    https://www.netbsd.org/~thorpej/rcorder-cache-diff-v4.txt

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | muc.lists.netbsd.tech.userlevel

csiph-web