Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #160088 > unrolled thread

nosh version 1.22

Back to article view | Back to linux.debian.user

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  nosh version 1.22 Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.com> - 2015-11-02 09:00 +0100
    nosh version 1.23 Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.com> - 2015-12-17 15:10 +0100
      nosh version 1.24 Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.com> - 2016-01-13 10:50 +0100
        nosh version 1.25 Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.com> - 2016-02-01 02:30 +0100
          nosh version 1.27 Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.com> - 2016-05-07 01:50 +0200

#160088 — nosh version 1.22

The nosh package is now up to version 1.22 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

There are several things in this release:

* a new binary package for FreeBSD
* improvements to the user-mode virtual terminal subsystem
* changes and additions to UCSPI tools
* log export to remote servers
* kqueue on Linux
* miscellany

a new binary package for FreeBSD
--------------------------------

There's now a debian-shims binary package for FreeBSD. This contains the 
heretofore not packaged invoke-rc.d and update-rc.d shim programs.  I 
decided to make this separate from the systemv-shims package because 
these areless general-purpose than those shims are.

The haltsys, fasthalt, fastpoweroff, and fastboot shims are now 
packaged, also.

improvements to the user-mode virtual terminal subsystem
--------------------------------------------------------

The console-fb-realizer now displays a mouse pointer sprite on the 
display, to reflect the position of the mouse, when an application has 
turned it on with the relevant control sequences. Mouse support via the 
evdev input subsystem on Linux is thus now fully implemented, including 
support for tablets that use absolute rather than relative positioning.

On the FreeBSD side, you can use sysmouse devices.  But this only 
permits relative positioning.  This is a limitation of sysmouse itself, 
as far as I can see.  A lot has to change, including the kernel, the 
protocol, and moused, to enable absolute positioned devices via 
sysmouse. Absolute positioning devices will therefore be supported using 
uhid devices.  Some of that is done already, but it's not complete yet.

Keyboard maps are now generated by the external configuration import 
subsystem from whatever one has in /usr/share/vt/keymaps , rather than 
being hardwired to a fixed set of countries.  In the absence of this 
directory (as will usually be the case on Linux operating systems), 
fallback U.S.A. and U.K. keyboard maps are generated.

This generation is worthy of note, as it exemplifies the mechanism that 
allows multiple BSD keyboard maps to be overlaid to make one generated 
map.  The fallback U.K. keyboard map is generated by taking the built-in 
U.S.A. keyboard map and applying a "us_to_uk" overlay map on top of it 
that only has the few differences between the U.S.A. International 
layout and the U.K. one.  (This currently produces the basic U.K. 
layout.  "U.K. Extended" should be a simple matter of another overlay 
that does the various Option+A -> a-acute mappings, but that's 
somethingfor the future.)  Similarly, versions of existing maps that 
swap Caps Lock and Control are generated by adding a simple overlay that 
does solely that.  Likewise, generated maps have an overlay applied that 
sets the Backspace key to the application-programmable DEC VT behaviour 
that console-terminal-emulator supports, that out-of-the-box BSD keymaps 
don't know anything about.

changes and additions to UCSPI tools
------------------------------------

For consistency, the UCSPI tools that supported a single --numeric 
option now support --numeric-host and --numeric-port options, for 
separately determining whether hosts and ports are taken to be names or 
just numbers.

There are now client-side tcp-socket-connect and udp-socket-connect 
tools, that open client sockets, connect them to servers, and then 
chain.  These adhere to the UCSPI conventions for inherited open file 
descriptors in client-side tools.

log export to remote servers
----------------------------

The new UCSPI clients were motivated by the new export-to-rsyslog 
command.  This is a daemon that expects to be invoked as a UCSPI client, 
connected to a remote RFC 5424/RFC 5426 ("rsyslog") server.  It 
maintains a set of "log cursors" that point to daemontools-stylelog 
directories. Tracking its position in the logs using those cursors, it 
sends new log information to the connected server.  In the usual nosh 
fashion, the filesystem is the database, and the "cursors" are just 
files and symbolic links.  The details are on the manual page.

In conjunction with the UCSPI clients, export-to-rsyslog thus makes a 
log remote export service.

This isn't intended to be the last word in such things. RFC 5426is 
unreliable, and RFC 5424 loses the microsecond and nanosecond 
information of TAI64N. But it demonstrates the idea and shows that this 
can be done in the daemontools world.  One can indeed export 
daemontools-stylelogsif one has (say) a suite of servers whose log data 
should be copied over, on the fly, to a centralized rsyslog server.  
There's room here for someone to take this idea and run further with it 
using something like RELP.

miscellany
----------

The several miscellaneous items include OpenLDAP services in the 
autoconfiguration subsystem and some tweaks to the /etc/fstab conversion 
on Linux to deal with records that don't explicitly say either 
read-write or read-only, resulting in undocumented behaviour in the 
Linux fstab parsing library.

On the subject of working around the behaviours of Linux libraries ...

libkqueue
---------

Those familiar with the development will know that Linux's libkqueue has 
been a perennial difficulty.  Its inaccessible private internal file 
descriptors are not marked close-on-exec, leaving open a security hole 
if libkqueue is used in a privileged process that forks off unprivileged 
children to run other programs.  NOTE_WRITE for EVFILT_VNODE isn't 
implemented correctly for directories.  And itbreaks when inotify events 
come through that have filenames in them.  Those familiar with the code 
will know that there was quite a lot of conditional compilation as a 
consequence, replacing libkqueue with individual hand-rolled mechanisms 
in those programs where libkqueue simply doesn't work or creates 
security weaknesses.  The final straw was a user reporting 
service-dt-scanner abending on Gentoo Linux when the scan directory is 
merely listed with "ls", which we eventually tracked down to libkqueue bugs.

No more.

I tried the route of patching libkqueue.  It was my preferred route.  
It's fairly easy to see where to add in the close-on-exec flags, for 
example.  The difficulty is in getting such things available both to 
users using Debian binary packages (on all of its various 
"stable"/"oldstable" flavours) and to users building from source on 
distributions that I don't have myself. In the end I took a step back 
and pondered whether libkqueue was even the right thing to be using in 
the first place.  After all, it's built to select from a multiplicity of 
implementations for several operating systems, using an internal 
abstraction layer,where the nosh toolset is only in fact ever using it 
for one.

So there's now an internal C++ kqueue/kevent library for Linux in the 
toolset, not ideal but "good enough" for the use that the nosh toolset 
needs from kqueue/kevent and doing the various things that it needs like 
close-on-exec, inotify with filenames and multiple events in one go, and 
proper NOTE_WRITE for directories; and the conditional code, the 
individual hand-rolled mechanisms (apart from one), and the binary 
package dependenciesfrom libkqueue are now gone.

[toc] | [next] | [standalone]

#161836 — nosh version 1.23

The nosh package is now up to version 1.23 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

There is one major item in this release.

* I've adjusted console-fb-realizer's keyboard handling on BSD to use 
USB directly.

There are a few more minor changes.

* I've upgraded the version of clang++ that is used to build the binary 
packages to 3.8.0. This should have no visible effect whatsoever.  (-:
* The handling of the DECDA2 control sequence by 
console-terminal-emulator now copes with what vim sends.  (What vim 
sends isn't what my DEC VT tests had been checking.)
* convert-systemd-units now inserts uses of the ionice and chrt 
chain-loading commands on Linux into the generated service bundles.

Mostly this is a clearing the decks release in the hope that I will be 
able to do some more work on the remaining few FreeBSD conversions 
before the new year.

USB keyboard support
====================

The keyboard handling is a change to using the USB HID devices 
(/dev/uhid*) on FreeBSD in preference to (but not forcibly instead of) 
the ATA keyboard protocol.  In part this is in order to handle the 
"consumer" keys that USB has.  In part this is in order to handle the 
extra keys that one finds on 106-key, 107-key, and 109-key keyboards and 
on some numeric keypads (such as the ABNT2 thousands-separator key).  In 
part it's to remove an extra layer of the user-space virtual terminal 
system that can be outwith the kernel.  In part it's to match the USB 
mouse capability from version 1.22 of the toolset.

Please note that the structure of kbdmap files has changed slightly, to 
accomodate mappings for "consumer" keys, to reposition the entries for 
some of the 106/107/109-key keyboards' extra keys, and to cover all of 
the function key gymnastics that vim can accept.  The 
/etc/system-control/convert/ system should automatically re-convert your 
VT kbd files into the new format.As part of this, I've moved the mapping 
for the Euro symbol in the fallback U.K. layout (as generated on Linux 
in the absence of VT kbd files).  It used to be level 3 shift on the 
[eE] key in prior versions of the toolset.  Almost all real U.K. 
keyboards nowadays have it engraved as level 3 shift on the [4$] key, 
and that's where it now is.

Also note that I'm still working on this.  There might be further 
changes in 1.24.  I've found a U.K. keyboard with two [#~] keys (at A00 
and C12), and I need to check out whether this actually employs what I 
had thought to be an error in the USB HID usage tables (distinct usages 
for "\|" and "Europe1") and had corrected, or whether this is a quite 
mad keyboard that simply has two "Europe1" keys (or two "\|" keys).  
Also, I've ordered an ABNT2 and a Japanese USB keyboard, and hope to do 
some testing with them, which may prompt further tweaks.  (I really 
wanted to buy a Leadership 4530 keyboard.  They seem to be out of stock 
in a lot of places.)

[toc] | [prev] | [next] | [standalone]

#162983 — nosh version 1.24

The nosh package is now up to version 1.24 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

Minor items in this release include:

* A fix for BSD keyboard layout import, that makes both "duml" and 
"ddia" be U+0308 for now.  Technically, diaeresis and umlaut are 
distinguishable in Unicode decomposed forms (using U+034F).  But for 
now, everything is simple unadorned combining diaeresis.
* A few more service bundles, for DBMail and for sudo (which in its 
vanilla form puts its timestamp files in /var/lib instead of /var/run 
and needs a cleanup service -- see Debian Bug #786555).
* Use of rtprio and idprio when converting system service units on 
FreeBSD/PC-BSD.
* Improvements to the framebuffer video mode selection in user-space 
virtual terminals for FreeBSD/PC-BSD.  It now comes up in the same 
display size as on Debian Linux on my test machines.
* Doco and other fixes from user feedback on version 1.23.  (I've 
already begun some further VirtualBox host adjustments, as we discussed, 
for 1.25.)

There is one major item in this release.

PC-BSD 10.2
===========

Until now, I'd been testing on a PC-BSD system that had been upgraded, 
with various contortions, from version 9.  This was still using UFS 
filesystems, listed in /etc/fstab; which the external configuration 
import subsystem had been happily importing to native service bundles.  
Over Christmastide I installed a PC-BSD 10.2 system from scratch, 
discovering some interesting oddities.  These included installation 
failing if you tell it that you are in the United Kingdom using a U.K. 
keyboard (PC-BSD Bug #12986); and the GRUB menu editor, as configured by 
the installer, operating in a truly eye-stretching 46 column by 28 row 
mode (by my count), and not displaying the underscore character correctly.

The important thing to know is that PC-BSD has for some time (at least 
since 2013) been ZFS-only, as far as installation goes.  (One can of 
course mount other filesystem types after installation.)  As Henry Ford 
might have said "Any customer can install to any filesystem type that xe 
likes, as long as it is ZFS.".  The result is that if installing from 
scratch one gets a whole load of ZFS datasets, and an empty (save for 
/proc and swap) /etc/fstab file.

So the major push for version 1.24 has been to get the configuration 
import system to deal with this, which it now does.  It will create 
mount services for all ZFS mounts, enable the ones that are "on", give 
them an inter-service ordering, and deal with the special-casing for the 
root (which the installer, oddly, marks as not automatically mounted, 
even though it of course is).

Alongside this, a large chunk of the remaining NetBSD rc.d services, 
from the on-going project to entirely replace them, have been crossed 
off the list.  These include mfs for /tmp, static networking and static 
ARP, pefs, serial port BPS and framing setup, ppp, rfcomm_pppd, 
persistent "entropy" for the randomness subsystem, and ipfw.  The 
progress of this work has been open from the start, and you can follow 
along on the roadmap WWW page.  Indeed, you can even join in, if you can 
convert any of the remaining few items.

There's more work to be done.  But I now have ZFS-only PC-BSD 10.2 
running nosh system-managed and service-managed.

Some notes for those eager to follow:

* Yes; I'm working on a pcdm service.  No; it doesn't help that it's 
undocumented.  Yes; that hoopla and palaver with forked subshells and 
multiple while loops calling sleep is exactly the sort of thing that 
proper service management is intended to obviate.

* If you have problems with devd, stale nologin from previous boots, and 
other things that use /var/run, it's because the convert_varrun service 
isn't enabled and your system has not been thus or otherwise migrated to 
/run.  This will be properly enabled by a preset in the next version.  
Enable it and reboot.  Or just start it and reboot. Or just boot into 
rescue mode and turn /var/run into a symbolic link to /run yourself.

* No; the nosh-run-system-manager package doesn't work properly on 
PC-BSD, as it does on vanilla FreeBSD.  PC-BSD 10.2 doesn't use the 
FreeBSD boot loader, like my old upgraded installation of PC-BSD 9 did.  
It uses GRUB.  The PC-BSD people apparently plan to get rid of GRUB in 
the future, and use the FreeBSD loader once more.  So this problem goes 
away when GRUB does.  (-:  In the meantime, use 'set 
kFreeBSD.init_path="/sbin/service-manager"' in the GRUB configuration.

* The root-resizing subsystem that was new to FreeBSD version 10 still 
needs conversion.  But ironically it doesn't work on PC-BSD 10.2 in the 
first place.  It can only grow UFS volumes, and PC-BSD's root is not a 
UFS volume.

[toc] | [prev] | [next] | [standalone]

#163784 — nosh version 1.25

The nosh package is now up to version 1.25 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

As you may have noticed from discussions elsewhere, a new 
oom-kill-protect utility has snuck in at the last moment.  This takes 
Linux-style OOM Killer score adjustments (an integer between -1000 and 
1000), BSD-style binary YES/NO settings, or a special setting for 
querying the "oomprotect" environment variable; and tries to do the 
closest matching thing for each platform.  Details are in the manual, of 
course.  With this, the OOMScoreAdjust setting is now converted by the 
convert-systemd-units utility.

The local-syslogd, udp-syslogd, and syslogd service bundles make use of 
oom-kill-protect with the special environment variable setting in their 
run programs.  So FreeBSD bug #204741 is addressed in a more general 
fashion that can be easily used in other service bundles. "rcctl set 
syslogd oomprotect YES" and "rcctl set syslogd oomprotect NO" can be 
used to turn OOM Killer protection on and off.

Other things in this version include:

* More configuration import utilities, covering ip6addrctl, webcamd, and 
NFS settings.
* A fix for a problem with configuration import on Linux in version 1.24.
* Two minor utilities for querying the fstab database, get-mount-what 
and get-mount-where, needed by the configuration import for mdconfig 
(but generally usable).
* New binary "run-" packages for OpenSSH server, syslog on a local 
socket, and klog.
* The new syslog and klog packages provide the Debian package manager's 
virtual package names "linux-kernel-log-daemon" and "system-log-daemon" 
(per Debian Bug #67604).

As can be seen from the roadmap, we are nearing the end of the rc.d 
conversion for FreeBSD.  Additions in this release include nfsserver, 
gptboot, rtadvd, virecover, and pcdm.  Almost all of mdconfig is 
actually done, bar some after/before orderings.

* 
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/roadmap.html#FreeBSDrc.d

[toc] | [prev] | [next] | [standalone]

#167546 — nosh version 1.27

The nosh package is now up to version 1.27 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

In fact, it is soon to be version 1.28.  This is a somewhat delayed 
notice for 1.27, because I forgot to send out the notices for versions 
1.27 and 1.26 after updating the WWW site.

As can be seen from the roadmap, we are at the point in rc.d conversion 
for FreeBSD/PC-BSD where it's actually easier to count the things that 
remain unconverted.  Discounting the PC-BSD Active Directory services 
and a handful of suspect FreeBSD services (such as growfs, which doesn't 
apply to ZFS in the first place) the remaining things to be converted 
can be counted on the fingers of one hand.  The external configuration 
import mechanism has gained the ability to handle stf, atabridge, 
mdconfig, and a few others. There are also a whole bunch more service 
bundles: cross-platform, for Linux, and for BSD.

* 
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/roadmap.html#FreeBSDrc.d

The OOM Killer avoidance measures from version 1.25 are now employed in 
the PostgreSQL service bundle as well.

The systemd service unit conversion tool has gained a whole load of 
NUMA-related extensions: NUMAInterleave, NUMAMemBind, NUMACPUNodeBind, 
NUMAPhysCPUBind, NUMALocalAlloc, and NUMAPreferred.  These it translates 
into the equivalent invocations of the numactl chain-loading utility.

It has also gained a couple of minor fixes and tweaks.  The %m 
substitution now works, and service bundles comprising FIFOs or AF_LOCAL 
sockets are now created so that they are ordered after any relevant 
filesystem mount services.

By request, the nosh Guide has gained a whole chapter of cheatsheets, 
giving quick one-liner pointers to some common tasks. The chapter is 
divided into three sections: chain loading, logging, and service 
management.  The service management division is subdivided into 
daemontools-style commands, systemd-style commands, OpenBSD-style 
commands, SMF-style commands, and common commands. The chain loading 
division gives a number of the more common commands used in 
chain-loading run scripts (and whereever else one might want to use them).

There have been improvements in static network setup, including fixes 
for some bugs in static_arp and static_ndp and a more cross-platform 
replacement for the static-networking service.

The nosh-bundles package now supplies several aliases for services, 
which are just plain old symbolic links.  So (for example) one can 
address the CUPS service as either org.cups.cupsd or just plain cupsd.

Things to look forward to in version 1.28 already include: more service 
bundles; another chain-loading utility; a major revision to MySQL and 
MariaDB service bundling, to reflect the pushes by their own developers 
to obviate their rc scripts and the mysql-safe command and just run 
mysqld directly under service management using the tools provided by the 
service management system; and a change relevant to the all-important 
linux_logo command.  (-:

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web