Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #270320 > unrolled thread

About dash as sh

Back to article view | Back to linux.debian.user

Contents

  About dash as sh Ilya Kazakevich <ilya.kazakevich@jetbrains.com> - 2024-06-21 00:30 +0200
    Re: About dash as sh Mike Castle <dalgoda+debian@gmail.com> - 2024-06-21 09:00 +0200
    Re: About dash as sh Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-06-21 12:20 +0200
      Re: About dash as sh Greg Wooledge <greg@wooledge.org> - 2024-06-21 13:30 +0200
        Re: About dash as sh Nicolas George <george@nsup.org> - 2024-06-21 13:50 +0200
          Re: About dash as sh Greg Wooledge <greg@wooledge.org> - 2024-06-21 14:00 +0200
            Re: About dash as sh Mike Castle <dalgoda+debian@gmail.com> - 2024-06-21 18:50 +0200
              Re: About dash as sh Greg Wooledge <greg@wooledge.org> - 2024-06-21 19:20 +0200
          Re: About dash as sh Stefan Monnier <monnier@iro.umontreal.ca> - 2024-06-22 01:30 +0200
            Re: About dash as sh Nicolas George <george@nsup.org> - 2024-06-24 08:20 +0200
              Re: About dash as sh Jeffrey Walton <noloader@gmail.com> - 2024-06-24 08:40 +0200
                [OFFTOPIC] Re: About dash as sh Stefan Monnier <monnier@iro.umontreal.ca> - 2024-06-24 15:00 +0200
    Re: About dash as sh Ilya Kazakevich <ilya.kazakevich@jetbrains.com> - 2024-06-26 20:20 +0200
      Re: About dash as sh Max Nikulin <manikulin@gmail.com> - 2024-06-28 05:00 +0200

#270320 — About dash as sh

Hello,

I've recently come across a bug in dash.

https://lore.kernel.org/dash/CAMQsgbSZnEac=ETYnR6a_ysnAysaHThwY03pnoDxC=p5FqtAag@mail.gmail.com/T

This issue is known for 7 years:
https://groups.google.com/g/linux.debian.bugs.dist/c/c6kRE-fhyuM

Fix is 18 months old, but unfortunately not released yet. Hence, we
have this issue even in sid (as I understand).


As this bug doesn't exist in bash I started thinking: why does Debian
use dash at all (not like RH for example, which uses `bash` for `sh)?

It turned out that 27 years ago there were 2 arguments:
1) Speed: bash is much larger and slower, and boot time was affected.
2) Posix compatibility.

The former argument is probably not so important now since Debian uses
`systemd` (no more sh scripts) and, honestly, I can't imagine how bash
could be a bottleneck for anything in 2024 (if you have such
scenarios, please share).

The latter is also a little bit strange as aforenamed bug breaks POSIX
compatibility (yes, stable Debian has a bug that breaks POSIX).

Having two shells (one for scripting and other one for interactive)
might lead to some other inconsistencies (one code-base is usually
more consistent than two).

With all of that I am pretty sure there should be some reason why dash
is still `sh` in Debian, and I must be missing something.

So, what is the reason?

Thank you,

Ilya.

[toc] | [next] | [standalone]

#270338

bash is still 10x larger than dash:
$ ls -l /bin/[bd]ash
-rwxr-xr-x 1 root root 1265648 Apr 23  2023 /bin/bash
-rwxr-xr-x 1 root root  125640 Jan  5  2023 /bin/dash

I would not be surprised if that impacts things like initrd and other
resource constrained environments.


Generally speaking, standards require multiple implementations.  So
having dash and bash leads to more consistency, not less.

Folks have been using different shells for interactive and scripting
usage for years.  Just check in with anyone who uses csh for their
interactive shell.  That does not mean they write scripts in csh.

Bash is known to have deviations from POSIX compliance, even in POSIX
mode (though much fewer than I remember from the last time I bothered
checking).

On the other hand, it appears that POSIX is in the middle of a cycle
introducing new shell features and Bash is actively implementing them.
I have no idea if dash is doing similar.  So it could be that, in a
year or two, Bash is more compliant than dash.

mrc

[toc] | [prev] | [next] | [standalone]

#270340

On 21 Jun 2024 00:28 +0200, from ilya.kazakevich@jetbrains.com (Ilya Kazakevich):
> [...] honestly, I can't imagine how bash
> could be a bottleneck for anything in 2024 (if you have such
> scenarios, please share).

Debian doesn't target only desktops and servers, where your assertion
is quite possibly correct. It's equally supported on comparatively
very low-powered systems; consider for example a low-RAM, perhaps also
slow-storage armel system.

Also, Debian doesn't target only this-year's systems. My own desktop
system uses a CPU which wasn't even top of the line when I put the
computer together over a decade ago now, and I like that Debian runs
well on it without requiring me to buy a new computer every few years.
(The one I had before this one reached a similar age before it broke
beyond the point of reasonably fixing it by replacing individual
parts.) Not only does it save money, it also saves on limited physical
resources and results in significantly less e-waste. Yes, _one_
computer may be relatively inconsequential, but in aggregate it does
add up.

-- 
Michael Kjörling                     🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

[toc] | [prev] | [next] | [standalone]

#270343

The original message began with the assertion that the OP had run
across a bug in dash, and gave two URLs, with no description of the bug
or the impact it was having on their life.

I read one of the URLs, and the bug is rather obscure.  It involves a
second script embedded inside a here document inside the first script,
with the second script being passed to an interpreter process on stdin.
I'm not surprised that nobody knew about the bug for many years.

So, having found one obscure bug in dash, the OP decided that the
best solution is to change Debian's years-old /bin/sh policy.

This ignores the fact that all shells, including bash, have *lots*
of bugs in them.  Switching /bin/sh to another shell would simply be
trading one set of bugs for a different set.

Given that Debian *originally* used bash as /bin/sh, and made a
conscious decision to switch that default to dash several years ago,
it would take an overwhelmingly strong reason to revert that change.
"I found an obscure bug in dash that affects me and one other person"
is probably not strong enough, especially when the bug has been fixed
upstream (albeit not in a released version yet??).

A more productive course of action would be to open a Debian bug report
against dash, describe the issue and how it affects you, point to the
upstream patch, and hope that a patched version of dash makes it into
trixie.

[toc] | [prev] | [next] | [standalone]

#270344

Greg Wooledge (12024-06-21):
> The original message began with the assertion that the OP had run
> across a bug in dash, and gave two URLs, with no description of the bug
> or the impact it was having on their life.
> 
> I read one of the URLs, and the bug is rather obscure.  It involves a
> second script embedded inside a here document inside the first script,
> with the second script being passed to an interpreter process on stdin.
> I'm not surprised that nobody knew about the bug for many years.

The purported bug boils down to this: if you pipe to a non-interactive
shell a command and data for that command, then the non-interactive
shell might read more than just the command as part of its input
buffering and leave less or nothing as data to the command itself.

It is indeed a bug, since the standard says:

  When the shell is using standard input and it invokes a command that
  also uses standard input, the shell shall ensure that the standard
  input file pointer points directly after the command it has read when
  the command begins execution.

But I consider this clause is misguided, it should apply only when the
input is a tty. Relying on it is a terrible idea.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#270345

On Fri, Jun 21, 2024 at 13:44:35 +0200, Nicolas George wrote:
> Greg Wooledge (12024-06-21):
> > The original message began with the assertion that the OP had run
> > across a bug in dash, and gave two URLs, with no description of the bug
> > or the impact it was having on their life.
> > 
> > I read one of the URLs, and the bug is rather obscure.  It involves a
> > second script embedded inside a here document inside the first script,
> > with the second script being passed to an interpreter process on stdin.
> > I'm not surprised that nobody knew about the bug for many years.
> 
> The purported bug boils down to this: if you pipe to a non-interactive
> shell a command and data for that command, then the non-interactive
> shell might read more than just the command as part of its input
> buffering and leave less or nothing as data to the command itself.
> 
> It is indeed a bug, since the standard says:
> 
>   When the shell is using standard input and it invokes a command that
>   also uses standard input, the shell shall ensure that the standard
>   input file pointer points directly after the command it has read when
>   the command begins execution.

I understood the bug as described.  I've been doing shell stuff for
a while now, and I've picked up a few bits of knowledge here and there.
I still claim that it's an obscure corner case that most script writers
will never encounter.

That's why I find it frustrating when someone claims that this bug is
so severe that Debian has to *change their policy* without even describing
how this bug is affecting them in real life.

> But I consider this clause is misguided, it should apply only when the
> input is a tty. Relying on it is a terrible idea.

I think the POSIX wording is there primarily because of here documents,
and people doing what I can only *guess* is similar to what the OP of
this thread wanted to do -- embedding layers of scripts inside scripts
using here documents.

I doubt that the wording was intended only for input coming from
terminals.

[toc] | [prev] | [next] | [standalone]

#270347

On Fri, Jun 21, 2024 at 4:57 AM Greg Wooledge <greg@wooledge.org> wrote:
> That's why I find it frustrating when someone claims that this bug is
> so severe that Debian has to *change their policy* without even describing
> how this bug is affecting them in real life.

I did not feel like the OP was saying the bug was that bad and the
policy needed to change, but as a starting point to ask why it is
still the policy after 27 years.

mrc

[toc] | [prev] | [next] | [standalone]

#270348

On Fri, Jun 21, 2024 at 09:43:52 -0700, Mike Castle wrote:
> On Fri, Jun 21, 2024 at 4:57 AM Greg Wooledge <greg@wooledge.org> wrote:
> > That's why I find it frustrating when someone claims that this bug is
> > so severe that Debian has to *change their policy* without even describing
> > how this bug is affecting them in real life.
> 
> I did not feel like the OP was saying the bug was that bad and the
> policy needed to change, but as a starting point to ask why it is
> still the policy after 27 years.

Are you unaware that it *changed*?

Here's a quote from <https://wiki.debian.org/Shell> which was the
first place I could find it:

  In all releases up to and including DebianLenny, Bash was the default
  non-interactive shell. Beginning with DebianSqueeze, Debian uses Dash
  (the Debian Almquist shell) as the target of the /bin/sh symlink.

Debian made a *conscious choice* to switch /bin/sh from bash to dash.
The OP of this thread is requesting that Debian should reverse this
and change *back* to bash, because of one bug, which affects a very
small number of scripts.

Furthermore:

  From DebianSqueeze to DebianBullseye, it was possible to select Bash
  as the target of the /bin/sh symlink by running dpkg-reconfigure
  dash. However, as of DebianBookworm, this is no longer supported.

So, the OP is not only asking for a reversion of the policy decision
that was made, but for a reinvestment of the time and resources that
would be required to support this new-but-really-old policy.  The
resources to support /bin/sh -> bash have already been discontinued.

[toc] | [prev] | [next] | [standalone]

#270351

>   When the shell is using standard input and it invokes a command that
>   also uses standard input, the shell shall ensure that the standard
>   input file pointer points directly after the command it has read when
>   the command begins execution.
>
> But I consider this clause is misguided, it should apply only when the
> input is a tty.

And if it's not a tty, you get some kind of Undefined Behavior?
I don't think I'd like that because I don't think the benefit would be worth
the UB troubles.

> Relying on it is a terrible idea.

I'd tend to agree.


        Stefan

[toc] | [prev] | [next] | [standalone]

#270422

Stefan Monnier (12024-06-21):
> And if it's not a tty, you get some kind of Undefined Behavior?

Knowing that “undefined behavior” is just an expression invented by C
standards authors to make “we make no guarantee about it, use it at your
own risk” sound more scary, I do not think it is a severe problem.

> I don't think I'd like that because I don't think the benefit would be worth
> the UB troubles.

The reasoning is the other way around: this feature should not be used,
and therefore the trouble of standardizing it is a waste of time.

The reason this feature should not be used is that it is exceptional. It
does not work with scripting languages that read their whole script
before running. It does not work with chained commands: “(head -n 2 >
/tmp/1 ; head -n 4 > /tmp/2)” will not put the next four lines in the
/tmp/2 file. None of the standard shell utilities makes any effort to
control buffering, and doing so would either change their semantics or
ruin their performances. Only the shell itself has this constraint.

So in order to use this guarantee properly, you need to be absolutely
sure you are in the exact case it covers. And odds are you will realize
it does not work because you added something in between that does
buffering.

Better just design your script in a way that does not rely on buffering
subtleties.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#270423

On Mon, Jun 24, 2024 at 2:16 AM Nicolas George <george@nsup.org> wrote:
>
> Stefan Monnier (12024-06-21):
> > And if it's not a tty, you get some kind of Undefined Behavior?
>
> Knowing that “undefined behavior” is just an expression invented by C
> standards authors to make “we make no guarantee about it, use it at your
> own risk” sound more scary, I do not think it is a severe problem.

Do shells suffer UB? I always thought that was a C thing.

When I encounter UB in C, I drop into inline assembler since asm does
not suffer C's undefined behavior.

Jeff

[toc] | [prev] | [next] | [standalone]

#270429 — [OFFTOPIC] Re: About dash as sh

> Do shells suffer UB? I always thought that was a C thing.

UB is a standard concept when defining programming languages.
Most if not all programming languages have some form of UB or another in
some part of the spec.  C is special w.r.t to UB in two ways:

- UB is not relegated to corner cases that virtually never happen (like
  the UB recommended by Jeffrey for sh), as is usually the case.
  Instead almost all programs in actual use rely on UB during their
  normal execution and in many cases it's somewhere between hard and
  impossible to avoid.
- Modern C compilers like to optimize code based on the assumption that
  UB never happens.

The combination of the two makes it particularly entertaining.


        Stefan

[toc] | [prev] | [next] | [standalone]

#270502

Hello,

Thank you for your answers.

My intention was to understand if decisions about dash are still
valid, not to tell Debian to switch back to bash of course.

Speaking about "bug or not":

This bug was confirmed by author:
https://lore.kernel.org/dash/Zm5y3du0C2MHdhSR@gondor.apana.org.au/

And here is the commit that fixes it:
https://git.kernel.org/pub/scm/utils/dash/dash.git/commit/?id=5f094d08c5bcee876191404a4f3dd2d075571215

As one might see, it has a POSIX quote in its commit message, so this
behaviour is a part of POSIX.

>>>
When the shell is using standard input and it invokes a command that
 also uses standard input, the shell shall ensure that the standard
 input file pointer points directly after the command it has read when
 the command begins execution. It shall not read ahead in such a manner
 that any characters intended to be read by the invoked command are
 consumed by the shell (whether interpreted by the shell or not) or
 that characters that are not read by the invoked command are not seen
 by the shell.
>>>

I am not the only one who found it:
https://michael-prokop.at/blog/2017/05/18/debugging-a-mystery-ssh-causing-strange-exit-codes/

There is even a Debian bug that clearly reports POSIX conformance issue:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=862907


So the bug, indeed, exists, but this is not a reason to switch to bash
as any software has bugs.

Even if it weren't covered by POSIX I wouldn't call it "undefined
behaviour", but "unspecified".
I am not sure if the term "undefined" is valid for shells (I guess
not) but in the C world that means compiler might do anything (think
"0 != 0" or the whole program optimized to "return;") in any place
since the app is ill.

https://en.cppreference.com/w/c/language/behavior

Many modern languages do not have this concept (although they have
race conditions, unspecified behaviour etc).


Speaking about different shells for interactive and scripting usage
and csh as an example.

AFAIR csh was originally created to substitute Bourne shell and Joy
tried to make it to be more C-like both for interactive and
programming usage (why study new syntax if everyone in the UNIX world
speaks C?)

Bourne sh was pretty weak back then: it didn't even have job control,
so csh was like a shiny future.
Later, SysV gained Korn Shell which was as powerful as csh, but
backward compatible with bourne shell.

So, BSD had csh and SysV had ksh (which was not portable to BSD due to license).

POSIX claimed that there should be a Bourne shell named `sh`, and BSD
had to have both: csh and Bourne shell to be POSIX compatible.

People started to write scripts in csh, but later on it was considered
bad practice.
There is a well-known document called "Csh Programming Considered
Harmful" from 1996
https://harmful.cat-v.org/software/csh

Csh exists on some BSDs because of lots of people who studied it long
ago and do not want to change old behaviours.

Do they regret this decision? I do not know, but OpenBSD, for example,
adopted pdksh (which is public domain ksh) as it is free, light
(lighter than bash), non-GPL and compatible with Bourne shell. They do
not use csh

https://misc.openbsd.narkive.com/Ekowx1ya/why-ksh

I doubt that any shell was created to be "scripting only": even ash
wasn't. The reason it didn't have history was that the author believed
that this should be implemented by tty driver called atty:
https://groups.google.com/g/comp.sources.unix/c/A6cnyKX-Gq4/m/dGKOOmXndCcJ

>>>
. History. It seems to me that the csh history mechanism is
mostly a response to the deficiencies of UNIX terminal I/O.
Those of you running 4.2 BSD should try out atty (which I am
posting to the net at the same time as ash) and see if you
still want history.
>>>
But he changed it soon after:
https://www.in-ulm.de/~mascheck/various/ash/#44bsdalpha


Having two implementations of the same standard is good for standard
(if standard has single implementation only, I wouldn't call it real
standard), but might produce compatibility issues between
implementations and might be impractical.

Anyway, I now see that dash is here for the good reason as Debian
could still be run on 15 years old hardware, tiny computers etc.

Ilya.

[toc] | [prev] | [next] | [standalone]

#270530

On 27/06/2024 01:16, Ilya Kazakevich wrote:
> My intention was to understand if decisions about dash are still
> valid, not to tell Debian to switch back to bash of course.

I am almost sure there are lengthy threads on Ubuntu and Debian mailing 
lists. Perhaps summaries are deeply buried. I was not interested in it, 
so I have in my notes just a couple of links related to transition to dash

https://askubuntu.com/questions/976485/what-is-the-point-of-sh-being-linked-to-dash
> Another benefit of dash is that it only relies on libc (the core system
> library) whereas bash also relies on terminal support libraries (it
> can't start without them, even to run a script); this means that dash
> has a better chance to keep working on a broken system.
2017-11-14 23:48:23Z

It may be realistic to get the dash bug fixed in next Debian release 
even without a new dash version. Try that the patch does not break any 
tests and file Debian and Ubuntu bugs.

Perhaps I have read some discussion of shell behavior that might be 
caused by this bug.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web