Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #268013 > unrolled thread

Re: bash parameter expansion "doesn't like" dots?

Back to article view | Back to linux.debian.user

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: bash parameter expansion "doesn't like" dots? Albretch Mueller <lbrtchx@gmail.com> - 2024-03-04 01:00 +0100
    Re: bash parameter expansion "doesn't like" dots? David Wright <deblis@lionunicorn.co.uk> - 2024-03-04 02:10 +0100
      Re: bash parameter expansion "doesn't like" dots? John Crawley <john@bunsenlabs.org> - 2024-03-04 04:10 +0100
        Re: bash parameter expansion "doesn't like" dots? David Wright <deblis@lionunicorn.co.uk> - 2024-03-04 21:30 +0100
          Re: bash parameter expansion "doesn't like" dots? John Crawley <john@bunsenlabs.org> - 2024-03-05 03:00 +0100
            Re: bash parameter expansion "doesn't like" dots? Greg Wooledge <greg@wooledge.org> - 2024-03-05 03:10 +0100
              Re: bash parameter expansion "doesn't like" dots? John Crawley <john@bunsenlabs.org> - 2024-03-05 03:30 +0100
                Re: bash parameter expansion "doesn't like" dots? Greg Wooledge <greg@wooledge.org> - 2024-03-05 04:00 +0100
                  Re: bash parameter expansion "doesn't like" dots? David <bouncingcats@gmail.com> - 2024-03-05 06:30 +0100
                    Re: bash parameter expansion "doesn't like" dots? Max Nikulin <manikulin@gmail.com> - 2024-03-05 12:40 +0100
              Re: bash parameter expansion "doesn't like" dots? Max Nikulin <manikulin@gmail.com> - 2024-03-05 03:40 +0100
                Re: bash parameter expansion "doesn't like" dots? John Crawley <john@bunsenlabs.org> - 2024-03-05 04:00 +0100

#268013 — Re: bash parameter expansion "doesn't like" dots?

 bash doesn't seem to like dots too close to brackets:

 echo "${_VAR//[^0-9a-zA-Z.,_-]/}"

 works fine.

 lbrtchx

On 3/3/24, Albretch Mueller <lbrtchx@gmail.com> wrote:
> _VAR="admissions.piedmont.edu_files?trackid=wnm:1980&PDFfiller=what-is-the-second-fundamental-theorem-of-calculus(1).pdf"
>
> echo "${_VAR//[^a-zA-Z0-9_-]/}"
>
> echo "${_VAR//[^a-zA-Z0-9_-.]/}"
>

[toc] | [next] | [standalone]

#268014

On Sun 03 Mar 2024 at 17:58:53 (-0600), Albretch Mueller wrote:
>  bash doesn't seem to like dots too close to brackets:
> 
>  echo "${_VAR//[^0-9a-zA-Z.,_-]/}"
> 
>  works fine.
> 
> On 3/3/24, Albretch Mueller <lbrtchx@gmail.com> wrote:
> > _VAR="admissions.piedmont.edu_files?trackid=wnm:1980&PDFfiller=what-is-the-second-fundamental-theorem-of-calculus(1).pdf"
> >
> > echo "${_VAR//[^a-zA-Z0-9_-]/}"
> >
> > echo "${_VAR//[^a-zA-Z0-9_-.]/}"
                             ↑↑↑

That's a range, except that it isn't because it's written backwards.
Check for yourself by testing with 9-0 instead of 0-9.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#268016

On 04/03/2024 10:07, David Wright wrote:
> On Sun 03 Mar 2024 at 17:58:53 (-0600), Albretch Mueller wrote:
>>   bash doesn't seem to like dots too close to brackets:
>>
>>   echo "${_VAR//[^0-9a-zA-Z.,_-]/}"
>>
>>   works fine.
>>
>> On 3/3/24, Albretch Mueller <lbrtchx@gmail.com> wrote:
>>> _VAR="admissions.piedmont.edu_files?trackid=wnm:1980&PDFfiller=what-is-the-second-fundamental-theorem-of-calculus(1).pdf"
>>>
>>> echo "${_VAR//[^a-zA-Z0-9_-]/}"
>>>
>>> echo "${_VAR//[^a-zA-Z0-9_-.]/}"
>                               ↑↑↑
> 
> That's a range, except that it isn't because it's written backwards.
> Check for yourself by testing with 9-0 instead of 0-9.
> 
> Cheers,
> David.
> 
So the problem isn't about dots, but the handling of the - which has to go last if it isn't to be treated as a range marker.

https://www.gnu.org/software/grep/manual/html_node/Character-Classes-and-Bracket-Expressions.html
says:

‘-’
     represents the range if it’s not first or last in a list or the ending point of a range. To make the ‘-’ a list item, it is best to put it last.

-- 
John

[toc] | [prev] | [next] | [standalone]

#268030

On Mon 04 Mar 2024 at 11:51:29 (+0900), John Crawley wrote:
> On 04/03/2024 10:07, David Wright wrote:
> > On Sun 03 Mar 2024 at 17:58:53 (-0600), Albretch Mueller wrote:
> > >   bash doesn't seem to like dots too close to brackets:
> > > 
> > >   echo "${_VAR//[^0-9a-zA-Z.,_-]/}"
> > > 
> > >   works fine.
> > > 
> > > On 3/3/24, Albretch Mueller <lbrtchx@gmail.com> wrote:
> > > > _VAR="admissions.piedmont.edu_files?trackid=wnm:1980&PDFfiller=what-is-the-second-fundamental-theorem-of-calculus(1).pdf"
> > > > 
> > > > echo "${_VAR//[^a-zA-Z0-9_-]/}"
> > > > 
> > > > echo "${_VAR//[^a-zA-Z0-9_-.]/}"
> >                               ↑↑↑
> > 
> > That's a range, except that it isn't because it's written backwards.
> > Check for yourself by testing with 9-0 instead of 0-9.
> > 
> So the problem isn't about dots,

It shouldn't be, as there's nothing special about ".", though there's
the mystery of what was said in the OP's quoted post, which we're not
privy to.

> but the handling of the - which has to go last if it isn't to be treated as a range marker.

First or last in the set. I prefer last, because "]" can only go first
in the set to be a matchable character.

> https://www.gnu.org/software/grep/manual/html_node/Character-Classes-and-Bracket-Expressions.html
> says:
> 
> ‘-’
>     represents the range if it’s not first or last in a list or the ending point of a range. To make the ‘-’ a list item, it is best to put it last.

Well, here's a clue as to where the trouble might have arisen.
Pattern matching in the shell is not the same as in grep: the
rules are different, but similar enough to confuse.

Which shell also matters. The OP appears to be using ^ to negate,
but ! has the advantage that it will be understood in bash and dash.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#268044

On 05/03/2024 05:27, David Wright wrote:
> Pattern matching in the shell is not the same as in grep: the
> rules are different, but similar enough to confuse.
Grep uses regular expressions, while the shell is usually globs. (I have no experience of shells other than dash and bash though.)
Bash can compare with regexes using the =~ operator [[ $A =~ $B ]] ...

> Which shell also matters. The OP appears to be using ^ to negate,
> but ! has the advantage that it will be understood in bash and dash.

I think ^ has been deprecated recently. I failed to find a reference on the web just now though.

Testing with dash on Bullseye:
$ v=string
$ echo ${v#*[!s]}
ring
$ echo ${v#*[^s]}
ring

But on Bookworm:
$ v=string
$ echo ${v#*[!s]}
ring
$ echo ${v#*[^s]}
tring

Now the ^ is being treated as just a list member.

With Bash the ^ still seems to be treated as a negator on Bookworm.

So yes, we should be switching to !

-- 
John

[toc] | [prev] | [next] | [standalone]

#268045

On Tue, Mar 05, 2024 at 10:49:34AM +0900, John Crawley wrote:
> On 05/03/2024 05:27, David Wright wrote:
> > Which shell also matters. The OP appears to be using ^ to negate,
> > but ! has the advantage that it will be understood in bash and dash.
> 
> I think ^ has been deprecated recently. I failed to find a reference on the web just now though.

POSIX specifies that ! is the negation character in glob ranges, largely
because ^ used to be a synonym for | in the old Bourne shell (for the
benefit of keyboards that didn't have a convenient | character).

The use of ^ as a glob negation is a bash extension.  It's nice for
people who may not even realize that they're supposed to be using !
because they learned regular expressions first.

So, ^ isn't "deprecated".  It's just not portable to sh.

[toc] | [prev] | [next] | [standalone]

#268046

On 05/03/2024 11:02, Greg Wooledge wrote:
> On Tue, Mar 05, 2024 at 10:49:34AM +0900, John Crawley wrote:
>> On 05/03/2024 05:27, David Wright wrote:
>>> Which shell also matters. The OP appears to be using ^ to negate,
>>> but ! has the advantage that it will be understood in bash and dash.
>>
>> I think ^ has been deprecated recently. I failed to find a reference on the web just now though.
> 
> POSIX specifies that ! is the negation character in glob ranges, largely
> because ^ used to be a synonym for | in the old Bourne shell (for the
> benefit of keyboards that didn't have a convenient | character).
> 
> The use of ^ as a glob negation is a bash extension.  It's nice for
> people who may not even realize that they're supposed to be using !
> because they learned regular expressions first.
> 
> So, ^ isn't "deprecated".  It's just not portable to sh.

^ worked as a negator in dash character classes up to Bullseye though, so something has changed recently. That's what my web searching failed to find...

-- 
John

[toc] | [prev] | [next] | [standalone]

#268049

On Tue, Mar 05, 2024 at 11:24:11AM +0900, John Crawley wrote:
> ^ worked as a negator in dash character classes up to Bullseye though, so something has changed recently. That's what my web searching failed to find...

It looks like dash doesn't have up-to-date documentation on its changes.
There's a ChangeLog file in the upstream Git repository's top level
directory[1] (shipped as changelog.gz in the Debian package), but the most
recent entry in it is dated 2014-11-17.

We might *guess* that this change was made to make dash more strict
about POSIX minimalism (removing extensions), but without documentation
we can't do more than guess about motives.

[1] https://git.kernel.org/pub/scm/utils/dash/dash.git/tree/

[toc] | [prev] | [next] | [standalone]

#268054

On Tue, 5 Mar 2024 at 02:59, Greg Wooledge <greg@wooledge.org> wrote:
> On Tue, Mar 05, 2024 at 11:24:11AM +0900, John Crawley wrote:

> > ^ worked as a negator in dash character classes up to Bullseye though, so something has changed recently. That's what my web searching failed to find...
>
> It looks like dash doesn't have up-to-date documentation on its changes.
> There's a ChangeLog file in the upstream Git repository's top level
> directory[1] (shipped as changelog.gz in the Debian package), but the most
> recent entry in it is dated 2014-11-17.
>
> We might *guess* that this change was made to make dash more strict
> about POSIX minimalism (removing extensions), but without documentation
> we can't do more than guess about motives.
>
> [1] https://git.kernel.org/pub/scm/utils/dash/dash.git/tree/

A bit more info:
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028002

Previous discussion on debian-user:
  https://lists.debian.org/debian-user/2023/04/msg00559.html

[toc] | [prev] | [next] | [standalone]

#268058

> On Tue, 5 Mar 2024 at 02:59, Greg Wooledge wrote:
>>
>> We might *guess* that this change was made to make dash more strict
>> about POSIX minimalism (removing extensions), but without documentation
>> we can't do more than guess about motives.

The motivation is to avoid difference in behavior when compiled with 
internal fnmatch implementation and with glibc

https://lore.kernel.org/dash/e341e41f-8c32-b6e4-8be3-8f94fae2677c@gigawatt.nl/
Re: possible wrong behaviour with patterns using a quoted ^ at the start 
of a bracket expression. Wed, 12 Jan 2022 17:20:54 +0000
> This bug (you're right that it's a bug) is specific to builds that use 
> fnmatch(). In dash itself, ^ is always assumed as a literal. In builds 
> with --disable-fnmatch you get correct results.

On 05/03/2024 12:22, David wrote:
> A bit more info:
>    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028002

Unfortunately this bug has been left without a response from the 
maintainer. The release notes mentions the issue:

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.html#dash-circumflex
https://bugs.debian.org/1034344

[toc] | [prev] | [next] | [standalone]

#268047

On 05/03/2024 09:02, Greg Wooledge wrote:
> On Tue, Mar 05, 2024 at 10:49:34AM +0900, John Crawley wrote:
>>
>> I think ^ has been deprecated recently. I failed to find a reference on the web just now though.
> 
> So, ^ isn't "deprecated".  It's just not portable to sh.

Running shellcheck on a *sh* script with a [^s] glob gives 
https://www.shellcheck.net/wiki/SC3026
"In POSIX sh, ^ in place of ! in glob bracket expressions is undefined."
with some links. There is no warning in the case of a #!/bin/bash script.

[toc] | [prev] | [next] | [standalone]

#268048

On 05/03/2024 11:36, Max Nikulin wrote:
> On 05/03/2024 09:02, Greg Wooledge wrote:
>> On Tue, Mar 05, 2024 at 10:49:34AM +0900, John Crawley wrote:
>>>
>>> I think ^ has been deprecated recently. I failed to find a reference on the web just now though.
>>
>> So, ^ isn't "deprecated".  It's just not portable to sh.
> 
> Running shellcheck on a *sh* script with a [^s] glob gives https://www.shellcheck.net/wiki/SC3026
> "In POSIX sh, ^ in place of ! in glob bracket expressions is undefined."
> with some links. There is no warning in the case of a #!/bin/bash script.
> 
> 
Thanks! Shellcheck also says:
"Dash used to support [^c] when compiled with fnmatch and glob from glibc, but it was considered as a bug and fixed in version 0.5.12."

That's the version of dash which arrived in Debian Bookworm.
-- 
John

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web