Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #224795 > unrolled thread

delimiters with more than one character? ...

Back to article view | Back to linux.debian.user

Contents

  delimiters with more than one character? ... Albretch Mueller <lbrtchx@gmail.com> - 2020-07-14 15:00 +0200
    Re: delimiters with more than one character? ... Alex Mestiashvili <amestia@rsh2.donotuse.de> - 2020-07-14 15:00 +0200
      Re: delimiters with more than one character? ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-14 15:10 +0200
        Re: delimiters with more than one character? ... Nicolas George <george@nsup.org> - 2020-07-14 15:10 +0200
        Re: delimiters with more than one character? ... Alex Mestiashvili <amestia@rsh2.donotuse.de> - 2020-07-14 15:20 +0200
    Re: delimiters with more than one character? ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-14 15:10 +0200
    Re: delimiters with more than one character? ... Nicolas George <george@nsup.org> - 2020-07-14 15:20 +0200
    Re: delimiters with more than one character? ... Nicolas George <george@nsup.org> - 2020-07-14 17:30 +0200
      Re: delimiters with more than one character? ... Albretch Mueller <lbrtchx@gmail.com> - 2020-07-15 12:30 +0200
        Re: delimiters with more than one character? ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-15 14:00 +0200
        Re: delimiters with more than one character? ... David Wright <deblis@lionunicorn.co.uk> - 2020-07-15 14:30 +0200
        Re: delimiters with more than one character? ... Bob Weber <bobrweber@gmail.com> - 2020-07-15 14:40 +0200
          Re: delimiters with more than one character? ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-15 14:50 +0200
            Re: delimiters with more than one character? ... Bob Weber <bobrweber@gmail.com> - 2020-07-15 16:20 +0200
              Re: delimiters with more than one character? ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-15 16:30 +0200
              Re: delimiters with more than one character? ... The Wanderer <wanderer@fastmail.fm> - 2020-07-15 16:30 +0200
              Re: delimiters with more than one character? ... Nicolas George <george@nsup.org> - 2020-07-15 17:50 +0200
          Re: delimiters with more than one character? ... Nicolas George <george@nsup.org> - 2020-07-15 15:10 +0200
    Re: delimiters with more than one character? ... Bob Weber <bobrweber@gmail.com> - 2020-07-14 17:30 +0200
    Re: delimiters with more than one character? ... Alessandro Baggi <alessandro.baggi@gmail.com> - 2020-07-15 17:30 +0200
    Re: delimiters with more than one character? ... Eric S Fraga <e.fraga@ucl.ac.uk> - 2020-07-16 12:10 +0200
      Re: delimiters with more than one character? ... "Thomas Schmitt" <scdbackup@gmx.net> - 2020-07-16 13:40 +0200
        Re: delimiters with more than one character? ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-16 14:00 +0200
          Re: delimiters with more than one character? ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-16 16:00 +0200
            Re: delimiters with more than one character? ... Zenaan Harkness <zenaan@freedbms.net> - 2020-07-17 04:50 +0200
              Re: delimiters with more than one character? ... Nicolas George <george@nsup.org> - 2020-07-17 10:50 +0200
                Re: delimiters with more than one character? ... rhkramer@gmail.com - 2020-07-17 13:20 +0200
                  tmpfs is not a ramdisk (was: delimiters with more than one  character? ...) Nicolas George <george@nsup.org> - 2020-07-17 13:30 +0200
                    Re: tmpfs is not a ramdisk (was: delimiters with more than one  character? ...) David Wright <deblis@lionunicorn.co.uk> - 2020-07-17 18:20 +0200
                      Re: tmpfs is not a ramdisk (was: delimiters with more than one character? ...) rhkramer@gmail.com - 2020-07-17 20:30 +0200
                        Re: tmpfs is not a ramdisk (was: delimiters with more than one  character? ...) Zenaan Harkness <zenaan@freedbms.net> - 2020-07-18 07:20 +0200
                        Re: tmpfs is not a ramdisk (was: delimiters with more than one  character? ...) <tomas@tuxteam.de> - 2020-07-18 10:40 +0200
                  Re: delimiters with more than one character? ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-17 13:40 +0200
          Re: delimiters with more than one character? ... "Thomas Schmitt" <scdbackup@gmx.net> - 2020-07-16 16:00 +0200
          Re: delimiters with more than one character? ... Zenaan Harkness <zenaan@freedbms.net> - 2020-07-17 04:50 +0200
        Re: delimiters with more than one character? ... Eric S Fraga <e.fraga@ucl.ac.uk> - 2020-07-16 15:00 +0200
    Re: delimiters with more than one character? ... John Crawley <john@bunsenlabs.org> - 2020-08-05 08:30 +0200
      Re: delimiters with more than one character? ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-08-05 15:10 +0200
        Re: delimiters with more than one character? ... John Crawley <john@bunsenlabs.org> - 2020-08-06 05:00 +0200
          Re: delimiters with more than one character? ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-08-06 13:40 +0200
            Re: delimiters with more than one character? ... John Crawley <john@bunsenlabs.org> - 2020-08-07 03:10 +0200

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#224871

Try sed:

echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | sed -e 's/\\|/\n/g'

HTH,
eric

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.7 on Debian bullseye/sid

[toc] | [prev] | [next] | [standalone]

#224873

Hi,

Eric S Fraga wrote:
> echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | sed -e 's/\\|/\n/g'

I came that far too. One can wrap the result in quotation marks by adding
  | sed -e 's/^.*$/"&"/'

But how to get the lines into the array ?
I did not find much info in man bash how to do this with reasonable effort.
The mass assignment of array values seems not to respect "-quotes.

This further sequence nicely prints the index and the array element value:

  ; echo ) | \
  while read x
  do
    _S_AR[$i]="$x"
    echo $i "${_S_AR[$i]}"
    i=$(expr $i + 1)
  done

But afterwards the array elements are empty again.
The problem seems to be in the pipe before the "while".
Exporting or declaring -a _S_AR does not help.

So for now i only have this _nearly_ working:

  echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | sed -e 's/\\|/\n/g' >/tmp/x

  i=0
  while read x
  do
    _S_AR[$i]="$x"
    i=$(expr $i + 1)
  done </tmp/x

  rm /tmp/x

The leading and trailing blanks still get lost by "read", which for me would
be the final reason to abandon bash for the intended purpose and to begin
doing things in a programming language which gives me some more control.
(Old C programmer's wisdom:
 The last 5 percent of work in a convenience language need 95 percent of the
 project time.)


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#224874

On Thu, Jul 16, 2020 at 01:30:02PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> Eric S Fraga wrote:
> > echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | sed -e 's/\\|/\n/g'
> 
> I came that far too. One can wrap the result in quotation marks by adding
>   | sed -e 's/^.*$/"&"/'
> 
> But how to get the lines into the array ?

First, do not add quotation marks to it.  They just get in the way.

If you've got an input stream with one element per line that you want
to import into a bash array, you use the "mapfile" (or "readarray")
command.

mapfile -t myarray < <(blah | sed)

If it's a NUL-delimited stream, and you're in bash 4.4 or higher, you
can use mapfile -d '' myarray < <(...) instead.

If it's a NUL-delimited stream but you're in bash 4.3 or older, then
you have to use a loop.

myarray=()
while IFS= read -r -d '' i; do
  myarray+=("$i")
done < <(...)

[toc] | [prev] | [next] | [standalone]

#224879

On Thu, Jul 16, 2020 at 03:52:01PM +0200, Thomas Schmitt wrote:
> I tried
> 
>   echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | \
>   sed -e 's/\\|/\n/g' | \
>   mapfile -t _S_AR
> 
> with no visible effect on the array _S_AR. bash 4.3.30 and 5.0.11.

You can't use a pipeline, because each command in the pipeline runs in
a separate subshell.

> This works
> 
>   echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | sed -e 's/\\|/\n/g' >/tmp/x
> 
>   mapfile -t _S_AR < /tmp/x

Yes.  Temp files are fine.

> A workaround without temporary file would be
> 
>   x=" 34 + 45 \| abc \| 1 2 3 \| c\|123abc "
>   split=$(echo "$x" | sed -e 's/\\|/\n/g')
>   end=$(echo -n "$x" | sed -e 's/./+/g')"+"
> 
>   mapfile -t _S_AR <<$end
>   $split
>   $end

No... just use the process substitution that I showed you already.

mapfile -t myarray < <(some command)

Do you need an actual demonstration?

unicorn:~$ f() { printf '%s\n' ' one ' 'two' 't h r e e'; }
unicorn:~$ mapfile -t a < <(f)
unicorn:~$ declare -p a
declare -a a=([0]=" one " [1]="two" [2]="t h r e e")

[toc] | [prev] | [next] | [standalone]

#224892

On Thu, Jul 16, 2020 at 09:57:12AM -0400, Greg Wooledge wrote:
> On Thu, Jul 16, 2020 at 03:52:01PM +0200, Thomas Schmitt wrote:
> > I tried
> > 
> >   echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | \
> >   sed -e 's/\\|/\n/g' | \
> >   mapfile -t _S_AR
> > 
> > with no visible effect on the array _S_AR. bash 4.3.30 and 5.0.11.
> 
> You can't use a pipeline, because each command in the pipeline runs in
> a separate subshell.
> 
> > This works
> > 
> >   echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | sed -e 's/\\|/\n/g' >/tmp/x
> > 
> >   mapfile -t _S_AR < /tmp/x
> 
> Yes.  Temp files are fine.

And mount /tmp as ramdisk ftw.


> > A workaround without temporary file would be
> > 
> >   x=" 34 + 45 \| abc \| 1 2 3 \| c\|123abc "
> >   split=$(echo "$x" | sed -e 's/\\|/\n/g')
> >   end=$(echo -n "$x" | sed -e 's/./+/g')"+"
> > 
> >   mapfile -t _S_AR <<$end
> >   $split
> >   $end
> 
> No... just use the process substitution that I showed you already.
> 
> mapfile -t myarray < <(some command)
> 
> Do you need an actual demonstration?
> 
> unicorn:~$ f() { printf '%s\n' ' one ' 'two' 't h r e e'; }
> unicorn:~$ mapfile -t a < <(f)
> unicorn:~$ declare -p a
> declare -a a=([0]=" one " [1]="two" [2]="t h r e e")
> 

[toc] | [prev] | [next] | [standalone]

#224894

[Multipart message — attachments visible in raw view] — view raw

Zenaan Harkness (12020-07-17):
> And mount /tmp as ramdisk ftw.

tmpfs is not a ramdisk.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#224896

On Friday, July 17, 2020 04:45:00 AM Nicolas George wrote:
> Zenaan Harkness (12020-07-17):
> > And mount /tmp as ramdisk ftw.
> 
> tmpfs is not a ramdisk.

I'm not the op, but, in what sense do you say that?

I use tmpfs filesystems for files that either I want fast access to (ram speed) 
or want to make sure they disappear on reboot / power loss.

I have tried to do a little reading since I saw this, I see some references 
that say it is stored on RAM, some others that say it is stored in virtual 
memory and could be on swap.

Among other things, should I be worried that it won't disappear on power loss 
/ reboot?

[toc] | [prev] | [next] | [standalone]

#224897 — tmpfs is not a ramdisk (was: delimiters with more than one character? ...)

[Multipart message — attachments visible in raw view] — view raw

rhkramer@gmail.com (12020-07-17):
> I'm not the op, but, in what sense do you say that?

tmpfs is FILESYSTEM backed in memory. A ramdisk is a kind of DISK, i.e.
a block device, backed in memory and in which you can store a filesystem
if you want.

There are significant differences, like the fact that a file removed
from a tmpfs will really free the memory and make it available for the
system, while a file removed from a filesystem stored on ramdisk will
just have its data marked as available for the same filesystem.
(Nowadays, a ramdisk could implement the trim features, but that's
piling hacks on top of hacks.)

There are also performance differences, because filesystems are designed
to ensure consistency over catastrophic interruptions, which is
completely useless if the data itself does not survive catastrophic
interruptions.

> I use tmpfs filesystems for files that either I want fast access to (ram speed) 
> or want to make sure they disappear on reboot / power loss.
> 
> I have tried to do a little reading since I saw this, I see some references 
> that say it is stored on RAM, some others that say it is stored in virtual 
> memory and could be on swap.

I am pretty sure it can be on swap. If you want to be sure, just test:
create a tmpfs larger than your physical memory, and fill it with a big
file.

> Among other things, should I be worried that it won't disappear on power loss 
> / reboot?

Depends on what kind of disappear you want. If all you want is not to
have to worry about cleaning up, then you are good with tmpfs.

If you are worried that sensitive data may end up on disk, then encrypt
your swap with an ephemeral key.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#224903 — Re: tmpfs is not a ramdisk (was: delimiters with more than one character? ...)

On Fri 17 Jul 2020 at 13:22:12 (+0200), Nicolas George wrote:
> rhkramer@gmail.com (12020-07-17):

[…]

> > Among other things, should I be worried that it won't disappear on power loss 
> > / reboot?
> 
> Depends on what kind of disappear you want. If all you want is not to
> have to worry about cleaning up, then you are good with tmpfs.
> 
> If you are worried that sensitive data may end up on disk, then encrypt
> your swap with an ephemeral key.

In the absence of encryption, bear in mind that cleaning up only takes
place while the system boots up. Until then, everything remains on the
disk and can be examined merely by booting from, say, a USB stick instead.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#224906 — Re: tmpfs is not a ramdisk (was: delimiters with more than one character? ...)

On Friday, July 17, 2020 12:09:05 PM David Wright wrote:
> On Fri 17 Jul 2020 at 13:22:12 (+0200), Nicolas George wrote:
> > rhkramer@gmail.com (12020-07-17):
> […]
> 
> > > Among other things, should I be worried that it won't disappear on
> > > power loss / reboot?
> > 
> > Depends on what kind of disappear you want. If all you want is not to
> > have to worry about cleaning up, then you are good with tmpfs.
> > 
> > If you are worried that sensitive data may end up on disk, then encrypt
> > your swap with an ephemeral key.
> 
> In the absence of encryption, bear in mind that cleaning up only takes
> place while the system boots up. Until then, everything remains on the
> disk and can be examined merely by booting from, say, a USB stick instead.

Just to make sure I understand, you're talking about clean up of swap on an 
HDD partition, right?

As far as the tmpfs stuff in RAM, it basically disappears when the power decays 
(after a not too long delay) and barring some kind of NSA techniques to 
capture it?

(But, I guess I should think about encrypting swap :-(

[toc] | [prev] | [next] | [standalone]

#224914 — Re: tmpfs is not a ramdisk (was: delimiters with more than one character? ...)

On Fri, Jul 17, 2020 at 02:25:22PM -0400, rhkramer@gmail.com wrote:
> On Friday, July 17, 2020 12:09:05 PM David Wright wrote:
> > On Fri 17 Jul 2020 at 13:22:12 (+0200), Nicolas George wrote:
> > > rhkramer@gmail.com (12020-07-17):
> > […]
> > 
> > > > Among other things, should I be worried that it won't disappear on
> > > > power loss / reboot?
> > > 
> > > Depends on what kind of disappear you want. If all you want is not to
> > > have to worry about cleaning up, then you are good with tmpfs.
> > > 
> > > If you are worried that sensitive data may end up on disk, then encrypt
> > > your swap with an ephemeral key.
> > 
> > In the absence of encryption, bear in mind that cleaning up only takes
> > place while the system boots up. Until then, everything remains on the
> > disk and can be examined merely by booting from, say, a USB stick instead.
> 
> Just to make sure I understand, you're talking about clean up of swap on an 
> HDD partition, right?
> 
> As far as the tmpfs stuff in RAM, it basically disappears when the power decays 
> (after a not too long delay) and barring some kind of NSA techniques to 
> capture it?
> 
> (But, I guess I should think about encrypting swap :-(


PC snow-nerf: "You don't need encrytion if you've got nothing to hide, and unless you're visiting friends during the lock down which you must not do because it's like totally illegal, you've got nothing to hide."

Alpha nagger: "I don't have to be doing anything wrong, to want my privacy!"

(Hush now, you're only s'posed to talk about human rights, not actually live them .. right?)



If you're current RAM+swap is say 8GiB+8GiB, one alternative is to consider upgrading RAM to 16GiB and ditching swap entirely.  That could save your lazy butt from having to learn how to encrypt swap - and just so you know I do in fact want you to feel good about your choice, it will also mean a faster overall user experience ;D

[toc] | [prev] | [next] | [standalone]

#224917 — Re: tmpfs is not a ramdisk (was: delimiters with more than one character? ...)

[Multipart message — attachments visible in raw view] — view raw

On Fri, Jul 17, 2020 at 02:25:22PM -0400, rhkramer@gmail.com wrote:

[...]

> As far as the tmpfs stuff in RAM, it basically disappears when the power decays 
> (after a not too long delay) and barring some kind of NSA techniques to 
> capture it?

Unless (parts of) it have been swapped to disk...

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#224898

On Fri, Jul 17, 2020 at 07:13:04AM -0400, rhkramer@gmail.com wrote:
> On Friday, July 17, 2020 04:45:00 AM Nicolas George wrote:
> > Zenaan Harkness (12020-07-17):
> > > And mount /tmp as ramdisk ftw.
> > 
> > tmpfs is not a ramdisk.
> 
> I'm not the op, but, in what sense do you say that?

People who stress out over that typically fall into two categories:

1) "I don't want my Secret data being written to hardware, because They
   might be able to read it!!  And yes, my favorite color is a Secret.
   Everything I write is a Secret!  You can't be reading this!!"

2) "My disk will DIE if I write to it more than 37 times, so I must prevent
   ALL hardware writes!!"

This is in addition to the regular old anti-temp-filers:

3) "Temp files are evil and messy and I Do Not Like Them!!  But it's OK
   to use sed -i even though that uses a temp file, because I don't have
   to see it.  Yes, I do like the taste of sand.  Why do you ask?"

So, these people (especially the first two) get all worked up about
temp files, and will do incredible contortions and backflips to try
to avoid them.  Often, this breaks their script.  They may not even
care about that.  To some of these people, perceived cleanliness is
more important than the correct behavior of their software.

What they all forget is that *most* of the time, a temp file will never
be written to hardware, even if the file system is ext4 or whatever,
simply because the temp file is so short-lived.  If the program opens
the temp file, uses it, unlinks it and closes it, all within a very
short span of time, the kernel will not have had time to flush the
data to hardware.

That said, if you're writing programs for an environment where you *know*
a RAM-based file system should exist (like Debian with systemd, where
you can use $XDG_RUNTIME_DIR), you should consider using that for
your small temp files.

[toc] | [prev] | [next] | [standalone]

#224880

Hi,

Eric S Fraga wrote:
> See the "\n" (newline) in the replacement pattern?

Of course. Else there would be only one output line.


Greg Wooledge wrote:
> If you've got an input stream with one element per line that you want
> to import into a bash array, you use the "mapfile" (or "readarray")
> command.

I tried

  echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | \
  sed -e 's/\\|/\n/g' | \
  mapfile -t _S_AR

with no visible effect on the array _S_AR. bash 4.3.30 and 5.0.11.

This works

  echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | sed -e 's/\\|/\n/g' >/tmp/x

  mapfile -t _S_AR < /tmp/x

It also works interactively

  $ mapfile -t _S_AR
  1
  2
  3
  <Ctrl+D>
  $ echo "'${_S_AR[1]}'"
  '2'
  $

So the obstacle seems to be that variable settings made in a shell pipe
do not persist. (It's not a bug, but a feature ... )

A workaround without temporary file would be

  x=" 34 + 45 \| abc \| 1 2 3 \| c\|123abc "
  split=$(echo "$x" | sed -e 's/\\|/\n/g')
  end=$(echo -n "$x" | sed -e 's/./+/g')"+"

  mapfile -t _S_AR <<$end
  $split
  $end

Question with this hack is of course how long the $end string of the
here-document may become. Maybe one can come up with a shorter string
that surely does not occur as line in $split.


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#224891

On Thu, Jul 16, 2020 at 07:56:57AM -0400, Greg Wooledge wrote:
> On Thu, Jul 16, 2020 at 01:30:02PM +0200, Thomas Schmitt wrote:
> > Hi,
> > 
> > Eric S Fraga wrote:
> > > echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | sed -e 's/\\|/\n/g'
> > 
> > I came that far too. One can wrap the result in quotation marks by adding
> >   | sed -e 's/^.*$/"&"/'
> > 
> > But how to get the lines into the array ?
> 
> First, do not add quotation marks to it.  They just get in the way.
> 
> If you've got an input stream with one element per line that you want
> to import into a bash array, you use the "mapfile" (or "readarray")
> command.
> 
> mapfile -t myarray < <(blah | sed)
> 
> If it's a NUL-delimited stream, and you're in bash 4.4 or higher, you
> can use mapfile -d '' myarray < <(...) instead.
> 
> If it's a NUL-delimited stream but you're in bash 4.3 or older, then
> you have to use a loop.
> 
> myarray=()
> while IFS= read -r -d '' i; do
>   myarray+=("$i")
> done < <(...)


-This- goes straight into my cheatsheet!

TYVM,

[toc] | [prev] | [next] | [standalone]

#224875

On Thursday, 16 Jul 2020 at 13:30, Thomas Schmitt wrote:
> Eric S Fraga wrote:
>> echo " 34 + 45 \| abc \| 1 2 3 \| c\|123abc " | sed -e 's/\\|/\n/g'
>
> But how to get the lines into the array ?

See the "\n" (newline) in the replacement pattern?

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.7 on Debian bullseye/sid

[toc] | [prev] | [next] | [standalone]

#225869

On 2020-07-14 21:52, Albretch Mueller wrote:
>   I have a string delimited by two characters: "\|"
> 
>   _S=" 34 + 45 \| abc \| 1 2 3 \| c\|123abc "
> 
>   which then I need to turn into a array looking like:
> 
>    _S_AR=(
> " 34 + 45"
> " abc"
> " 1 2 3"
> " c"
> "123abc"
> )
> 
>    Is it possible to do such things in bash?

Yes, if you don't mind running a loop.
This method cuts off the first part of the string, up to the delimiter, 
and adds it to the array, then continues with what's left of the string 
until there's none left:
---
#!/bin/bash

_S=' 34 + 45 \| abc \| 1 2 3 \| c\|123abc '
del='\|'

arr=()
s=${_S}${del}
while [[ -n $s ]]
do
     arr+=( "${s%%"${del}"*}" )
     s=${s#*"${del}"}
done

declare -p arr
---
outputs:
declare -a arr=([0]=" 34 + 45 " [1]=" abc\\ " [2]=" 1 2 3 " [3]=" c" 
[4]="123abc ")

I've used this myself, so am eager to hear of any hidden snags. :)

(One already: if the delimiter is a repeated character which might also 
be the last in the last string fragment, then the loop never closes. 
Fairly rare?)

Originally found here:
https://www.tutorialkart.com/bash-shell-scripting/bash-split-string/

-- 
John

[toc] | [prev] | [next] | [standalone]

#225889

On Wed, Aug 05, 2020 at 03:04:32PM +0900, John Crawley wrote:
> This method cuts off the first part of the string, up to the delimiter, and
> adds it to the array, then continues with what's left of the string until
> there's none left:

Yes, that's a valid approach, although not one that most bash scripters
would choose, because it's hideously slow in bash, and doesn't "feel"
like it's in the "spirit" of shell scripting.

> ---
> #!/bin/bash
> 
> _S=' 34 + 45 \| abc \| 1 2 3 \| c\|123abc '
> del='\|'
> 
> arr=()
> s=${_S}${del}
> while [[ -n $s ]]
> do
>     arr+=( "${s%%"${del}"*}" )
>     s=${s#*"${del}"}
> done
> 
> declare -p arr
> ---
> outputs:
> declare -a arr=([0]=" 34 + 45 " [1]=" abc\\ " [2]=" 1 2 3 " [3]=" c"
> [4]="123abc ")

Your output doesn't match your input.  You've got an extra backslash
character in the [1] element.  Perhaps you tested with several different
inputs, and accidentally pasted the wrong output.

(It would have been good if the original poster's sample input string
had included some of the things you were probably testing, such as
standalone backslash and pipe characters.  As presented, the original
problem was pretty stupid.  Why even have a multi-character delimiter
when none of the individual characters in the delimiter appear in the
data?)

> I've used this myself, so am eager to hear of any hidden snags. :)
> 
> (One already: if the delimiter is a repeated character which might also be
> the last in the last string fragment, then the loop never closes. Fairly
> rare?)

OK... yeah, that would be a show-stopper, all right.  I was able
to reproduce the infinite loop using:

_S=' 34 + 45 || abc\ || 1|2 3 || c||123abc |'
del='||'
s=${_S}${del}
arr=()
while [[ -n $s ]]; do arr+=( "${s%%"${del}"*}" ); s=${s#*"${del}"}; done

(Don't do this in a window you care about, or on a system that can't
afford to run out of memory.  By the time I pressed Ctrl-C, the output
array had over 600,000 elements.)

The problem here is that the two expressions inside the loop fail to
identify the final delimiter correctly.  At the point where everything
fails, we have:

del='||'
s='123abc |||'

and:

$ printf '<%s>\n' "${s%%"${del}"*}"
<123abc >
$ printf '<%s>\n' "${s#*"${del}"}"
<|>

In other words, the expressions treat the *first* two pipes as the
delimiter, rather than the *last* two pipes.  The third pipe character
is therefore left behind in the string, causing the infinite loop.
(In addition, the final array element before the infinite loop is
incorrect.  It should be "123abc |", not "123abc ".)

I don't believe this approach can be made to work in all cases, because
appending the extra delimiter to the end of the string creates an
ambiguous input.  The ||| at the end could be "a pipe followed by
the delimiter", or "the delimiter followed by a pipe".  We humans
know that it's supposed to be the former, but the code in the loop
is resulting in the latter.

A different approach is needed.  The one that immediately springs to
mind for me is:

s='a || b || c ||| d |'
del='||'
arr=()
while [[ $s = *"$del"* ]]; do arr+=( "${s%%"$del"*}" ); s=${s#*"$del"}; done
arr+=( "$s" )

This is very similar to the flawed approach, but instead of appending
an extra delimiter to the input and looping until the input is empty,
we *check* for the existence of a delimiter in the input, and loop until
none is found.  Whatever is left over becomes the final array element.

This will be a little bit slower than the flawed code (probably), but I
believe it is free from infinite loops.  If you're not convinced,
you could add a loop counter, and abort when some arbitrary limit is
reached.

Since we're not modifying the input by appending an extra delimiter,
any ambiguity in the input string was put there by the original input
source, not created by our code.  In my sample input, the ||| substring
between c and d can be parsed as either "delimiter followed by pipe",
or "pipe followed by delimiter".  It is impossible to tell which one
is correct, because we have no external knowledge of the input string.
Therefore, either result must be acceptable.  If the humans who
provided this input don't care for the results they get, well, it's
their problem and not ours.

[toc] | [prev] | [next] | [standalone]

#225922

On 2020-08-05 22:06, Greg Wooledge wrote:
> On Wed, Aug 05, 2020 at 03:04:32PM +0900, John Crawley wrote:
>> This method cuts off the first part of the string, up to the delimiter, and
>> adds it to the array, then continues with what's left of the string until
>> there's none left:
> 
> Yes, that's a valid approach, although not one that most bash scripters
> would choose, because it's hideously slow in bash, and doesn't "feel"
> like it's in the "spirit" of shell scripting.

I likely don't have a sufficient grasp of that "spirit" to comment there :)
but as for "hideously slow", remain to be convinced. Bash string  
modifications seem to be pretty fast. Would very long strings requiring  
multiple runs of the loop slow it down excessively? (See below for  
stringx1024) With the OP's provided string, on my system:

time bash ./multi-delim.sh
declare -a arr=([0]=" 34 + 45 " [1]=" abc " [2]=" 1 2 3 " [3]=" c"  
[4]="123abc ")

real		0m0.004s
user	0m0.000s
sys		0m0.000s

Exact time varies from 1ms to 5ms, but it seems faster than python, perl  
or a bash script calling sed or awk would be.

> Your output doesn't match your input.  You've got an extra backslash
> character in the [1] element.  Perhaps you tested with several different
> inputs, and accidentally pasted the wrong output.

Thanks - that was indeed the case. Sorry.
( Tested with some other characters like * # ! Everything seems to be  
treated as a plain string, needing no escaping. )

>> I've used this myself, so am eager to hear of any hidden snags. :)
>>
>> (One already: if the delimiter is a repeated character which might also be
>> the last in the last string fragment, then the loop never closes. Fairly
>> rare?)
> 
> OK... yeah, that would be a show-stopper, all right.  I was able
> to reproduce the infinite loop...
(snip)
> A different approach is needed.  The one that immediately springs to
> mind for me is:
> 
> s='a || b || c ||| d |'
> del='||'
> arr=()
> while [[ $s = *"$del"* ]]; do arr+=( "${s%%"$del"*}" ); s=${s#*"$del"}; done
> arr+=( "$s" )
> 
> This is very similar to the flawed approach, but instead of appending
> an extra delimiter to the input and looping until the input is empty,
> we *check* for the existence of a delimiter in the input, and loop until
> none is found.  Whatever is left over becomes the final array element
> This will be a little bit slower than the flawed code (probably), but I
> believe it is free from infinite loops.

Nice! The run time is very similar.
Adding this:

for i in {1..10}
do
     _S+=$del$_S
done

to multipy the string by 1024 gave times of ~3.9s and ~4.0s.

I can't see any way the loop would fail to end. This will be my  
boilerplate if I ever need to do such a thing again...

> Since we're not modifying the input by appending an extra delimiter,
> any ambiguity in the input string was put there by the original input
> source, not created by our code.  In my sample input, the ||| substring
> between c and d can be parsed as either "delimiter followed by pipe",
> or "pipe followed by delimiter".  It is impossible to tell which one
> is correct, because we have no external knowledge of the input string.
> Therefore, either result must be acceptable.  If the humans who
> provided this input don't care for the results they get, well, it's
> their problem and not ours.

I guess there's little need to try to cope with delimiters that would be  
ambiguous in practice...

-- 
John

[toc] | [prev] | [next] | [standalone]

#225934

On Thu, Aug 06, 2020 at 11:57:47AM +0900, John Crawley wrote:
> I likely don't have a sufficient grasp of that "spirit" to comment there :)
> but as for "hideously slow", remain to be convinced. Bash string
> modifications seem to be pretty fast. Would very long strings requiring
> multiple runs of the loop slow it down excessively?

Absolutely.

> Adding this:
> 
> for i in {1..10}
> do
>     _S+=$del$_S
> done
> 
> to multipy the string by 1024 gave times of ~3.9s and ~4.0s.

For short strings, doing this sort of parsing in bash is fine.  But as
you can see, on large inputs, it does not scale well.  The fact that each
iteration makes a *copy* of nearly the entire remaining input doesn't
help matters, either -- right off the bat, that algorithm makes the run
time explode, even before you take into account bash's 100% interpreted
runtime slowness.

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | linux.debian.user

csiph-web