Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.shell > #22486 > unrolled thread

[ANN] KornShell 93u+m 1.0.0-beta.2

Back to article view | Back to comp.unix.shell

Contents

  [ANN] KornShell 93u+m 1.0.0-beta.2 Martijn Dekker <martijn@inlv.demon.nl> - 2021-12-17 04:38 +0100
    Re: [ANN] KornShell 93u+m 1.0.0-beta.2 Janis Papanagnou <janis_papanagnou@hotmail.com> - 2021-12-17 09:43 +0100
      Re: [ANN] KornShell 93u+m 1.0.0-beta.2 Martijn Dekker <martijn@inlv.demon.nl> - 2021-12-17 11:18 +0100
        Re: [ANN] KornShell 93u+m 1.0.0-beta.2 Spiros Bousbouras <spibou@gmail.com> - 2021-12-17 12:50 +0000
          Arithmetic trivia (Was: [ANN] KornShell 93u+m 1.0.0-beta.2) gazelle@shell.xmission.com (Kenny McCormack) - 2021-12-17 13:10 +0000
            Re: Arithmetic trivia (Was: [ANN] KornShell 93u+m 1.0.0-beta.2) Janis Papanagnou <janis_papanagnou@hotmail.com> - 2021-12-17 21:39 +0100
          Re: [ANN] KornShell 93u+m 1.0.0-beta.2 Martijn Dekker <martijn@inlv.demon.nl> - 2021-12-17 19:58 +0000
    Re: [ANN] KornShell 93u+m 1.0.0-beta.2 Janis Papanagnou <janis_papanagnou@hotmail.com> - 2021-12-18 15:07 +0100
      Re: [ANN] KornShell 93u+m 1.0.0-beta.2 Martijn Dekker <martijn@inlv.demon.nl> - 2021-12-19 21:00 +0000
        Re: [ANN] KornShell 93u+m 1.0.0-beta.2 Martijn Dekker <martijn@inlv.demon.nl> - 2021-12-21 19:12 +0000
        Re: [ANN] KornShell 93u+m 1.0.0-beta.2 Janis Papanagnou <janis_papanagnou@hotmail.com> - 2021-12-24 14:46 +0100

#22486 — [ANN] KornShell 93u+m 1.0.0-beta.2

Announcing: KornShell 93u+m 1.0.0-beta.2
https://github.com/ksh93/ksh

In May 2020, when every KornShell (ksh93) development project was
abandoned, development was rebooted in a new fork based on the last
stable AT&T version: ksh 93u+. This new fork is called ksh 93u+m as a
permanent nod to its origin. We're restarting it at version 1.0. Seven
months after the first beta, the second one is ready. Please test this
second beta and report any bugs you find, or help us fix known bugs.

We're now the default ksh93 in some OS distributions, at least Debian
and Slackware! Even though we don't think it's stable release quality
yet, the consensus seems to be that 93u+m is already much better than
the last AT&T release.

Main developers: Martijn Dekker, Johnothan King, hyenias

Contributors: Andy Fiddaman, Anuradha Weeraman, Chase, Gordon Woodhull,
Govind Kamat, Harald van Dijk, Lev Kujawski, Marc Wilson, Ryan Schmidt,
Sterling Jensen

HOW TO GET IT

Please download the source code tarball from our GitHub releases page:

	https://github.com/ksh93/ksh/releases

To build, follow the instructions in README.md or src/cmd/ksh93/README.

HOW TO GET INVOLVED

To report a bug, please open an issue at our GitHub page (see above).
Alternatively, email me at martijn@inlv.org with your report.
To get involved in development, read the brief policy information in
README.md and then jump right in with a pull request or email a patch.
See the TODO file in the top-level directory for a to-do list.

### MAIN CHANGES between 1.0.0-beta.1 and 1.0.0-beta.2 ###

New features in built-in commands:

- 'cd' now supports an -e option that, when combined with -P, verifies
   that $PWD is correct after changing directories; this helps detect
   access permission problems. See:
   https://www.austingroupbugs.net/view.php?id=253

- 'printf' now supports a -v option as in bash. This assigns formatted
   output directly to variables, which is very fast and will not strip
   final newline (\n) characters.

- The 'return' command, when used to return from a function, can now
   return any status value in the 32-bit signed integer range, like on
   zsh. However, due to a traditional Unix kernel limitation, $? is
   still trimmed to its least significant 8 bits whenever leaving a
   (sub)shell environment.

- 'test'/'[' now supports all the same operators as [[ (including =~,
   \<, \>) except for the different 'and'/'or' operators. Note that
   'test'/'[' remains deprecated due to its unfixable pitfalls;
   [[ ... ]] is recommended instead.

Shell language changes:

- Several improvements were made to the --noexec shell code linter.

- Arithmetic expressions in native ksh mode no longer interpret a
   number with a leading zero as octal in any context. Use 8#octalnumber
   instead (e.g. 8#400 == 256). Arithmetic expressions now also behave
   identically within and outside ((...)) and $((...)).

- POSIX compatibility mode fixes (only applicable with the --posix shell
   option on):
   - A leading zero is now consistently recognised as introducing an octal
     number in all arithmetic contexts.
   - $((inf)) and $((nan)) are now interpreted as regular variables.
   - The '.' built-in no longer runs ksh functions and now only runs
     files.

Bugs fixed:

- '.' and '..' are now once again completed by tab completion.

- If SIGINT is set to ignore, the interactive shell no longer exits on
   Ctrl+C.

- ksh now builds and runs on Apple's new M1 hardware.

- The 'return' and 'exit' commands no longer risk triggering actual
   signals by returning or exiting with a status > 256.

- Ksh no longer behaves badly when parsing a type definition command
   ('typeset -T' or 'enum') without executing it or when executing it in
   a subshell. Types can now safely be defined in subshells and defined
   conditionally as in 'if condition; then enum ...; fi'.

- Discipline functions, especially those applied to PS2 or .sh.tilde,
   will no longer crash your shell upon being interrupted or throwing an
   error.

- Fixed a bug that could corrupt output if standard output is closed
   upon initialising the shell.

- Fixed a bug in the [[ ... ]] compound command: the '!' logical
   negation operator now correctly negates another '!', e.g.,
   [[ ! ! 1 -eq 1 ]] now returns 0/true. Note that this has always been
   the case for 'test'/'['.

- Fixed SHLVL so that replacing ksh by itself (exec ksh) will not
   increase it.

- Arithmetic expressions are no longer allowed to assign out-of-range
   values to variables of types declared with enum.

- The 'time' keyword no longer makes the --errexit shell option
   ineffective.

- Various bugs in libcmd built-in commands (those bound to the
   /opt/ast/bin path by default) have been fixed.

- Various other crashing bugs have been fixed.

Fixes for the shcomp byte code compiler:

- shcomp is now able to compile scripts that define types using enum.

- shcomp now refuses to mess up your terminal by writing bytecode
   to it.

### MAIN CHANGES between ksh 93u+ 2012-08-01 and 93u+m 1.0.0-beta.1 ###

Hundreds of bugs have been fixed, including many serious/critical bugs.
This includes upstreamed patches from OpenSUSE, Red Hat, and Solaris, fixes
backported from the abandoned 93v- beta and ksh2020 fork, as well as many
new fixes from the community. See the NEWS file for more information, and
the git commit log for complete documentation of every fix. Incompatible
changes have been minimised, but not at the expense of fixing bugs. For a
list of potentially incompatible changes, see src/cmd/ksh93/COMPATIBILITY.

Though there was a "no new features, bugfixes only" policy, some new
features were found necessary, either to fix serious design flaws or to
complete functionality that was evidently intended, but not finished.
Below is a summary of these new features.

New command line editor features:

- The forward-delete and End keys are now handled as expected in the
   emacs and vi built-in line editors.

- In the vi and emacs line editors, repeat count parameters can now also
   be used for the arrow keys and the forward-delete key. E.g., in emacs
   mode, <ESC> 7 <left-arrow> will now move the cursor seven positions to
   the left. In vi control mode, this would be entered as: 7 <left-arrow>.

New shell language features:

- The &>file redirection shorthand (for >file 2>&1) is now available for
   all scripts and interactive sessions and not only for profile/login
   scripts, bringing ksh 93u+m in line with mksh, bash, and zsh.

- File name generation (a.k.a. pathname expansion, a.k.a. globbing) now
   never matches the special navigational names '.' (current directory)
   and '..' (parent directory). This change makes a pattern like .*
   useful; it now matches all hidden files (dotfiles) in the current
   directory, without the harmful inclusion of '.' and '..'.

- Tilde expansion can now be extended or modified by defining a
   .sh.tilde.get or .sh.tilde.set discipline function. This replaces a
   2004 undocumented attempt to add this functionality via a .sh.tilde
   command, which never worked and crashed the shell. See the manual for
   details on the new method.

New features in built-in commands:

- Usage error messages now show the --help/--man self-documentation options.

- Path-bound built-ins (such as /opt/ast/bin/cat) can now be executed by
   invoking the canonical path, so the following will now work as expected:
	$ /opt/ast/bin/cat --version
	  version         cat (AT&T Research) 2012-05-31

- 'command -x' now looks for external commands only, skipping built-ins.
   In addition, its xargs-like functionality no longer freezes the shell on
   Linux and macOS, making it effectively a new feature on these systems.

- 'redirect' now checks if all arguments are valid redirections before
   performing them. If an error occurs, it issues an error message instead
   of terminating the shell.

- 'suspend' now refuses to suspend a login shell, as there is probably no
   parent shell to return to and the login session would freeze.

- 'times' now gives high precision output in a POSIX compliant format.

- 'typeset' now gives an informative error message if an incompatible
   combination of options is given.

- 'whence -v/-a' now reports the location of autoloadable functions.

New features in shell options:

- A new --globcasedetect shell option is added on OSs where we can
   check for a case-insensitive file system (currently Windows/Cygwin,
   macOS, Linux and QNX 7.0+). When this option is turned on, file name
   generation (globbing), as well as file name tab completion on
   interactive shells, automatically become case-insensitive on file
   systems where the difference between upper and lower case is ignored
   for file names. This is transparently determined for each directory, so
   a path pattern that spans multiple file systems can be part
   case-sensitive and part case-insensitive.

- A new --nobackslashctrl shell option disables the special escaping
   behaviour of the backslash character in the emacs and vi built-in
   editors. Particularly in the emacs editor, this makes it much easier to
   go backward, insert a forgotten backslash into a command, and then
   continue editing without having your next cursor key replace your
   backslash with garbage. Note that Ctrl+V (or whatever other character
   was set using 'stty lnext') always escapes all control characters in
   either editing mode.

- A new --posix shell option has been added to ksh 93u+m that makes the
   ksh language more compatible with other shells by following the POSIX
   standard more closely. See the manual page for details. It is enabled by
   default if ksh is invoked as sh, otherwise it is disabled by default.

- Enhancement to -G/--globstar: symbolic links to directories are now
   followed if they match a normal (non-**) glob pattern. For example, if
   '/lnk' is a symlink to a directory, '/lnk/**' and '/l?k/**' now work as
   you would expect.

-- 
||    modernish -- harness the shell
||    https://github.com/modernish/modernish
||
||    KornShell lives!
||    https://github.com/ksh93/ksh

[toc] | [next] | [standalone]

#22489

On 17.12.2021 04:38, Martijn Dekker wrote:
> Announcing: KornShell 93u+m 1.0.0-beta.2
> https://github.com/ksh93/ksh

Great, thanks!

I'll have a look into the details soon, but this one irritated me;
but maybe I just misunderstood it:

> - The 'return' command, when used to return from a function, can now
>   return any status value in the 32-bit signed integer range, like on
>   zsh. However, due to a traditional Unix kernel limitation, $? is
>   still trimmed to its least significant 8 bits whenever leaving a
>   (sub)shell environment.

If in Kornshell I terminate a process with a signal I get a number
returned that is beyond the 8 bit range; e.g. a kill (with SIGTERM)
results in $? evaluating to 271.

Janis

[toc] | [prev] | [next] | [standalone]

#22496

Op 17-12-21 om 09:43 schreef Janis Papanagnou:
> I'll have a look into the details soon, but this one irritated me;
> but maybe I just misunderstood it:
> 
>> - The 'return' command, when used to return from a function, can now
>>    return any status value in the 32-bit signed integer range, like on
>>    zsh. However, due to a traditional Unix kernel limitation, $? is
>>    still trimmed to its least significant 8 bits whenever leaving a
>>    (sub)shell environment.
> 
> If in Kornshell I terminate a process with a signal I get a number
> returned that is beyond the 8 bit range; e.g. a kill (with SIGTERM)
> results in $? evaluating to 271.

Yes, that has not changed.

If a child process of ksh's is terminated by a signal, ksh doesn't 
inherit the exit status via the kernel as that process didn't exit 
normally. Instead, ksh detects that the process was terminated by a 
signal (see 'man 2 wait' for how that works) and sets $? to the signal 
number with the 9th bit set in the current process -- which is not 
subject to the kernel's 8-bit exit status limitation as that all happens 
in the parent process.

What the announcement item above means instead is this:

$ myfunc() { return 12345; }
$ myfunc; echo $?
12345
$ (myfunc); echo $?
57
$ ksh -c 'myfunc() { return 12345; }; myfunc'; echo $?
57

i.e., the exit status of a subshell or child shell that invokes 
'myfunc', and does nothing to overwrite $? before exiting, is the least 
significant 8 bits of 12345, i.e., 12345 % 0xFF == 210.

For consistency's sake, ksh 93u+m enforces this for virtual 
(non-forking) subshells as well, because a script should never be able 
to notice a difference between a virtual subshell and a subshell that 
forked a separate process due to executing something like 'ulimit'.

-- 
||    modernish -- harness the shell
||    https://github.com/modernish/modernish
||
||    KornShell lives!
||    https://github.com/ksh93/ksh

[toc] | [prev] | [next] | [standalone]

#22497

On Fri, 17 Dec 2021 11:18:26 +0100
Martijn Dekker <martijn@inlv.demon.nl> wrote:
> What the announcement item above means instead is this:
> 
> $ myfunc() { return 12345; }
> $ myfunc; echo $?
> 12345
> $ (myfunc); echo $?
> 57
> $ ksh -c 'myfunc() { return 12345; }; myfunc'; echo $?
> 57
> 
> i.e., the exit status of a subshell or child shell that invokes 
> 'myfunc', and does nothing to overwrite $? before exiting, is the least 
> significant 8 bits of 12345, i.e., 12345 % 0xFF == 210.

For the least significant 8 bits of 12345 the calculation you want is
12345 % 256 == 57 .Regardless ,  12345 % 0xFF == 105 .

[toc] | [prev] | [next] | [standalone]

#22498 — Arithmetic trivia (Was: [ANN] KornShell 93u+m 1.0.0-beta.2)

In article <JGDDG6C28aBHRfrQ6@bongo-ra.co>,
Spiros Bousbouras  <spibou@gmail.com> wrote:
>On Fri, 17 Dec 2021 11:18:26 +0100
>Martijn Dekker <martijn@inlv.demon.nl> wrote:
>> What the announcement item above means instead is this:
>> 
>> $ myfunc() { return 12345; }
>> $ myfunc; echo $?
>> 12345
>> $ (myfunc); echo $?
>> 57
>> $ ksh -c 'myfunc() { return 12345; }; myfunc'; echo $?
>> 57
>> 
>> i.e., the exit status of a subshell or child shell that invokes 
>> 'myfunc', and does nothing to overwrite $? before exiting, is the least 
>> significant 8 bits of 12345, i.e., 12345 % 0xFF == 210.
>
>For the least significant 8 bits of 12345 the calculation you want is
>12345 % 256 == 57 .Regardless ,  12345 % 0xFF == 105 .

If you want to use 255 (i.e., hex FF), then you have to use an 'and'
operator.

i.e., either of:

12345 % 256

or:

12345 & 255

-- 
If the automobile had followed the same development cycle as the
computer, a Rolls-Royce today would cost $100, get a million miles to
the gallon, and explode once every few weeks, killing everyone inside.

[toc] | [prev] | [next] | [standalone]

#22501 — Re: Arithmetic trivia (Was: [ANN] KornShell 93u+m 1.0.0-beta.2)

On 17.12.2021 14:10, Kenny McCormack wrote:
> 
> If you want to use 255 (i.e., hex FF), then you have to use an 'and'
> operator.
> i.e., either of:
> 12345 % 256
> or:
> 12345 & 255

Yep, just a typical well known careless mistake. In the same vein
as implementing the integer division
  x div 2   # x/2
by
  x shr 1   # x>>1
but writing by accident x>>2 (a careless mistake I made 35 years
ago in a test and still recall); a favorable reader will understand
the intention, but a computer will merciless execute it.

Janis

[toc] | [prev] | [next] | [standalone]

#22499

Op 17-12-21 om 12:50 schreef Spiros Bousbouras:
> On Fri, 17 Dec 2021 11:18:26 +0100
> Martijn Dekker <martijn@inlv.demon.nl> wrote:
>> What the announcement item above means instead is this:
>>
>> $ myfunc() { return 12345; }
>> $ myfunc; echo $?
>> 12345
>> $ (myfunc); echo $?
>> 57
>> $ ksh -c 'myfunc() { return 12345; }; myfunc'; echo $?
>> 57
>>
>> i.e., the exit status of a subshell or child shell that invokes
>> 'myfunc', and does nothing to overwrite $? before exiting, is the least
>> significant 8 bits of 12345, i.e., 12345 % 0xFF == 210.
> 
> For the least significant 8 bits of 12345 the calculation you want is
> 12345 % 256 == 57 .Regardless ,  12345 % 0xFF == 105 .

Indeed. Sorry. I was travelling and insufficiently caffeinated while 
writing that. What I *meant* was: 12345 & 0xFF == 57


-- 
||    modernish -- harness the shell
||    https://github.com/modernish/modernish
||
||    KornShell lives!
||    https://github.com/ksh93/ksh

[toc] | [prev] | [next] | [standalone]

#22503

On 17.12.2021 04:38, Martijn Dekker wrote:
> Announcing: KornShell 93u+m 1.0.0-beta.2
> https://github.com/ksh93/ksh
> [...]
> Please download the source code tarball from our GitHub releases page:
> 
>     https://github.com/ksh93/ksh/releases
> 
> To build, follow the instructions in README.md or src/cmd/ksh93/README.

I tried a build of the shell, and after a long seamless running compile
got an error. (I seem to recall that I detected it already in a previous
release.)

Note: in the logs below I replaced the actual top level directory by ~ .

+ ~/ksh/arch/linux.i386-64/bin/ar x -lsum sumlib.o
/usr/bin/ar: two different operation options specified
mamake [lib/libcmd]: *** exit code 1 making sumlib.o
mamake: *** exit code 1 making lib/libcmd
mamake: *** exit code 1 making all

Got the usage info from the packaged ar (no idea where the -U thing
comes from, and why a different path for ar is displayed)

~/ksh/arch/linux.i386-64/bin/ar --help
/usr/bin/ar: invalid option -- 'U'
Usage: /usr/bin/ar [emulation options] [-]{dmpqrstx}[abcDfilMNoPsSTuvV]
[--plugin <name>] [member-name] [count] archive-file file...
       /usr/bin/ar -M [<mri-script]

but it seems the call, instead of
  ar x -lsum sumlib.o
should probably rather be something like
  ar xl sum sumlib.o

An individual call reproduces the error

  ~/ksh/arch/linux.i386-64/bin/ar x -lsum sumlib.o
  /usr/bin/ar: two different operation options specified

while changing the separately started call just complains about the file

  ~/ksh/arch/linux.i386-64/bin/ar xl sum sumlib.o
  /usr/bin/ar: sum: No such file or directory

(which is understandable, since I wasn't executing the test call from
within the build system, probably being in a wrong path or environment
variables may be unset).

Is there any fix for that?

Janis

[toc] | [prev] | [next] | [standalone]

#22504

Op 18-12-21 om 14:07 schreef Janis Papanagnou:
> On 17.12.2021 04:38, Martijn Dekker wrote:
>> Announcing: KornShell 93u+m 1.0.0-beta.2
>> https://github.com/ksh93/ksh
>> [...]
>> Please download the source code tarball from our GitHub releases page:
>>
>>      https://github.com/ksh93/ksh/releases
>>
>> To build, follow the instructions in README.md or src/cmd/ksh93/README.
> 
> I tried a build of the shell, and after a long seamless running compile
> got an error. (I seem to recall that I detected it already in a previous
> release.)

Sorry, I'd forgotten about that.

> Note: in the logs below I replaced the actual top level directory by ~ .
> 
> + ~/ksh/arch/linux.i386-64/bin/ar x -lsum sumlib.o
> /usr/bin/ar: two different operation options specified
> mamake [lib/libcmd]: *** exit code 1 making sumlib.o
> mamake: *** exit code 1 making lib/libcmd
> mamake: *** exit code 1 making all
> 
> Got the usage info from the packaged ar (no idea where the -U thing
> comes from, and why a different path for ar is displayed)
> 
> ~/ksh/arch/linux.i386-64/bin/ar --help
> /usr/bin/ar: invalid option -- 'U'
> Usage: /usr/bin/ar [emulation options] [-]{dmpqrstx}[abcDfilMNoPsSTuvV]
> [--plugin <name>] [member-name] [count] archive-file file...
>         /usr/bin/ar -M [<mri-script]
[...]

The cause is probably this ar wrapper script:

     $ cat src/cmd/INIT/ar.linux.i386-64 

     : linux.i386-64 ar wrapper

     case $1 in
     *x*) /usr/bin/ar "$@" ;;
     *)   /usr/bin/ar U"$@" ;;
     esac

This gets copied into your package hierarchy as 
arch/linux.i386-64/bin/ar and put in $PATH.

If you delete the wrapper script in both places

     $ rm src/cmd/INIT/ar.linux.i386-64 arch/linux.i386-64/bin/ar

and then try 'bin/package make' again, does it build?

I have no idea what that U thing is for. The AT&T folks must have had 
some reason for adding it.

'man ar' on CentOS x86_64 documents:

        A number of modifiers (mod) may immediately follow the p
        keyletter, to specify variations on an operation's behavior:
[...]
        D   Operate in deterministic mode.  When adding files and the
            archive index use zero for UIDs, GIDs, timestamps, and use
            consistent file modes for all files.  When this option is
            used, if ar is used with identical options and identical
            input files, multiple runs will create identical output
            files regardless of the input files' owners, groups, file
            modes, or modification times.
[...]
        U   Do not operate in deterministic mode.  This is the inverse
            of the D modifier, above: added files and the archive index
            will get their actual UID, GID, timestamp, and file mode
            values.

            This is the default unless binutils was configured with
            --enable-deterministic-archives.

So the question is, why would we want to disable deterministic mode? It 
sounds like a good thing.

If you can confirm that deleting the wrapper script fixes your build 
failure, I'll get rid of it and see if anything breaks for anyone else.

(In fact in the long term I plan to get rid of all such wrapper scripts. 
It's a very flawed way of making a build work. Just overriding the 
default compiler or linker using the appropriate failures will disable 
the wrappers so the necessary flags aren't added. There are better ways 
of doing this.)

-- 
||    modernish -- harness the shell
||    https://github.com/modernish/modernish
||
||    KornShell lives!
||    https://github.com/ksh93/ksh

[toc] | [prev] | [next] | [standalone]

#22509

Op 19-12-21 om 21:00 schreef Martijn Dekker:
> If you can confirm that deleting the wrapper script fixes your build 
> failure, I'll get rid of it and see if anything breaks for anyone else.

In the meantime I got rid of that wrapper script and some others on the 
1.0 branch: https://github.com/ksh93/ksh/commit/24fc1bbc

-- 
||    modernish -- harness the shell
||    https://github.com/modernish/modernish
||
||    KornShell lives!
||    https://github.com/ksh93/ksh

[toc] | [prev] | [next] | [standalone]

#22519

On 19.12.2021 22:00, Martijn Dekker wrote:
> [...]
> If you delete the wrapper script in both places
> 
>     $ rm src/cmd/INIT/ar.linux.i386-64 arch/linux.i386-64/bin/ar
> 
> and then try 'bin/package make' again, does it build?

I'll try this and also your github changes after the holidays.

Thank you!

Janis

[toc] | [prev] | [standalone]

Back to top | Article view | comp.unix.shell

csiph-web