Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #247273 > unrolled thread

extract values from a string

Back to article view | Back to linux.debian.user

Contents

  extract values from a string wilson <info@bigcount.xyz> - 2022-04-16 04:40 +0200
    Re: extract values from a string Greg Wooledge <greg@wooledge.org> - 2022-04-16 05:10 +0200
    Re: extract values from a string Nicholas Geovanis <nickgeovanis@gmail.com> - 2022-04-16 18:10 +0200
      Re: extract values from a string <tomas@tuxteam.de> - 2022-04-16 18:20 +0200
        Re: extract values from a string Nicholas Geovanis <nickgeovanis@gmail.com> - 2022-04-16 18:50 +0200
        Re: extract values from a string Greg Wooledge <greg@wooledge.org> - 2022-04-16 19:30 +0200
      Re: extract values from a string Kent West <westk@acu.edu> - 2022-04-17 04:30 +0200
        Re: extract values from a string Greg Wooledge <greg@wooledge.org> - 2022-04-17 05:00 +0200
    Re: extract values from a string David Christensen <dpchrist@holgerdanske.com> - 2022-04-16 21:30 +0200
    Re: extract values from a string Ash Joubert <ash@transient.nz> - 2022-04-17 02:40 +0200
      Re: extract values from a string Ash Joubert <ash@transient.nz> - 2022-04-17 02:40 +0200
      Re: extract values from a string Greg Wooledge <greg@wooledge.org> - 2022-04-17 02:40 +0200
        Re: extract values from a string Ash Joubert <ash@transient.nz> - 2022-04-17 02:50 +0200

#247273 — extract values from a string

Hello

in shell script, how can I use regex to extract values from a string? 
maybe value types transformation should be required.


for instance the string: "black berry 12".
I want go get the name: black berry [String]
the price: 12 [Int]

I did this in other language such as java/scala:

scala> val regex = """(.+)\s+(\d+)""".r
val regex: scala.util.matching.Regex = (.+)\s+(\d+)

scala> str match {
      |   case regex(name,price) => (name,price.toInt)
      | }
val res2: (String, Int) = (black berry,12)


But sometimes for a simple task I won't want to write a java program.
so what's the shell solution?

Thanks

[toc] | [next] | [standalone]

#247275

On Sat, Apr 16, 2022 at 10:37:14AM +0800, wilson wrote:
> in shell script, how can I use regex to extract values from a string?

"Some people, when confronted with a problem, think ``I know, I'll use
regular expressions.'' Now they have two problems."

> for instance the string: "black berry 12".
> I want go get the name: black berry [String]
> the price: 12 [Int]

Ideally we would want to see more input samples.  Is the "price" always
the last input word, after the last whitespace character?  Is the "name"
always the set of words up to but not including the last chunk of
whitespace characters?

Let's assume that these guesses are true.

If the chunks of whitespace are always precisely ONE space character
apiece, then it's really easy:

string="black berry 12"
name=${string% *}
price=${string##* }

And that will work in sh (POSIX, not Bourne) or bash.  We can also allow
the whitespace to be either a space or a tab (or some other esoteric
characters) by replacing the literal space with a character class pattern:

string="black berry 12"
name=${string%[[:space:]]*}
price=${string##*[[:space:]]}

That's still POSIX sh compatible.

If the chunks might be MORE than one whitespace character apiece, then
we need something a bit uglier.  We could use bash with its extended
globs:

shopt -s extglob
string=$'black berry\t\t\t12'
price=${string##*[[:space:]]}
name=${string%%+([[:space:]])"$price"}

There are many other solutions, of course.

I also suggest <https://mywiki.wooledge.org/BashFAQ/100> as additional
reading.

[toc] | [prev] | [next] | [standalone]

#247297

[Multipart message — attachments visible in raw view] — view raw

On Fri, Apr 15, 2022, 9:37 PM wilson <info@bigcount.xyz> wrote:

> Hello
>
> in shell script, how can I use regex to extract values from a string?
> maybe value types transformation should be required.
>
>
> for instance the string: "black berry 12".
> I want go get the name: black berry [String]
> the price: 12 [Int]
>

So you have programming language skills. Look at the info-command-based doc
for bash. Read the sections about shell pattern-matching operators. Realize
that there are more than one type of that. For example the crippled
filename pattern matching on many commands, not really regular expressions.

Everything you want to do can be done. The syntax is clunkier than other
mechanisms.

I did this in other language such as java/scala:
>
> scala> val regex = """(.+)\s+(\d+)""".r
> val regex: scala.util.matching.Regex = (.+)\s+(\d+)
>
> scala> str match {
>       |   case regex(name,price) => (name,price.toInt)
>       | }
> val res2: (String, Int) = (black berry,12)
>
>
> But sometimes for a simple task I won't want to write a java program.
> so what's the shell solution?
>
> Thanks
>
>

[toc] | [prev] | [next] | [standalone]

#247298

[Multipart message — attachments visible in raw view] — view raw

On Sat, Apr 16, 2022 at 11:01:17AM -0500, Nicholas Geovanis wrote:
> On Fri, Apr 15, 2022, 9:37 PM wilson <info@bigcount.xyz> wrote:
> 
> > Hello
> >
> > in shell script, how can I use regex to extract values from a string?
> > maybe value types transformation should be required.
> >
> >
> > for instance the string: "black berry 12".
> > I want go get the name: black berry [String]
> > the price: 12 [Int]
> >
> 
> So you have programming language skills. Look at the info-command-based doc
> for bash. Read the sections about shell pattern-matching operators. Realize
> that there are more than one type of that. For example the crippled
> filename pattern matching on many commands, not really regular expressions.

FWIW, bash has an =~ operator for regex matching whithin [[ ]]. There are
even special variables to pick up the capturing matches.

Not portable, but arrays (which have been warmly recommended around here)
aren't, either.

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#247299

[Multipart message — attachments visible in raw view] — view raw

On Sat, Apr 16, 2022, 11:12 AM <tomas@tuxteam.de> wrote:

> On Sat, Apr 16, 2022 at 11:01:17AM -0500, Nicholas Geovanis wrote:
> > On Fri, Apr 15, 2022, 9:37 PM wilson <info@bigcount.xyz> wrote:
> >
> > > Hello
> > >
> > > in shell script, how can I use regex to extract values from a string?
> > > maybe value types transformation should be required.
> > >
> > >
> > > for instance the string: "black berry 12".
> > > I want go get the name: black berry [String]
> > > the price: 12 [Int]
> > >
> >
> > So you have programming language skills. Look at the info-command-based
> doc
> > for bash. Read the sections about shell pattern-matching operators.
> Realize
> > that there are more than one type of that. For example the crippled
> > filename pattern matching on many commands, not really regular
> expressions.
>
> FWIW, bash has an =~ operator for regex matching whithin [[ ]]. There are
> even special variables to pick up the capturing matches.
> Not portable, but arrays (which have been warmly recommended around here)
> aren't, either.
>

I didn't want to spoil the party and say this. But now my hand is forced :-)
You Mr Original Poster may find perl to be more congenial for regular
expression use especially. You should find it on every single platform you
find bash. Regardless of OS.

Cheers
> --
> t
>

[toc] | [prev] | [next] | [standalone]

#247300

On Sat, Apr 16, 2022 at 06:12:17PM +0200, tomas@tuxteam.de wrote:
> FWIW, bash has an =~ operator for regex matching whithin [[ ]]. There are
> even special variables to pick up the capturing matches.
> 
> Not portable, but arrays (which have been warmly recommended around here)
> aren't, either.

Yes, the =~ operator *could* be used for some kinds of parsing.  I just
didn't think it was the best choice for this specific problem.

[toc] | [prev] | [next] | [standalone]

#247315

[Multipart message — attachments visible in raw view] — view raw

On Sat, Apr 16, 2022 at 11:01 AM Nicholas Geovanis <nickgeovanis@gmail.com>
wrote:

> On Fri, Apr 15, 2022, 9:37 PM wilson <info@bigcount.xyz> wrote:
>
>> Hello
>>
>> in shell script, how can I use regex to extract values from a string?
>> maybe value types transformation should be required.
>>
>>
>> for instance the string: "black berry 12".
>> I want go get the name: black berry [String]
>> the price: 12 [Int]
>>
>
>
-- data.txt --
black berry 12
blue berry 14
raspberry 9
huckle berry hound 3
bare-knuckle sandwich 27

-- test.sh --
#!/bin/bash

file="data.txt"

while read -r line; do
   productID=$(awk -F' ' '{$NF=""; print $0}' <<< $line)
   price=$(awk -F' ' '{print $NF}' <<< $line)
   retail=$(($price * 2))
   printf "The price of \'$productID\' is \$$price. We will retail it for
\$$retail.\n"
done <$file


-- $./test.sh --
The price of 'black berry ' is $12. We will retail it for $24.
The price of 'blue berry ' is $14. We will retail it for $28.
The price of 'raspberry ' is $9. We will retail it for $18.
The price of 'huckle berry hound ' is $3. We will retail it for $6.
The price of 'bare-knuckle sandwich ' is $27. We will retail it for $54.

-- 
Kent West                    <")))><
Westing Peacefully - http://kentwest.blogspot.com

[toc] | [prev] | [next] | [standalone]

#247317

On Sat, Apr 16, 2022 at 09:27:52PM -0500, Kent West wrote:
> -- data.txt --
> black berry 12
> blue berry 14
> raspberry 9
> huckle berry hound 3
> bare-knuckle sandwich 27
> 
> -- test.sh --
> #!/bin/bash
> 
> file="data.txt"
> 
> while read -r line; do
>    productID=$(awk -F' ' '{$NF=""; print $0}' <<< $line)
>    price=$(awk -F' ' '{print $NF}' <<< $line)

Two awks...

>    retail=$(($price * 2))
>    printf "The price of \'$productID\' is \$$price. We will retail it for
> \$$retail.\n"
> done <$file
> 
> 
> -- $./test.sh --
> The price of 'black berry ' is $12. We will retail it for $24.
> The price of 'blue berry ' is $14. We will retail it for $28.
> The price of 'raspberry ' is $9. We will retail it for $18.
> The price of 'huckle berry hound ' is $3. We will retail it for $6.
> The price of 'bare-knuckle sandwich ' is $27. We will retail it for $54.

... and you still left trailing whitespace on the product names.

Also, your printf is wrong.  You're injecting the variable contents into
the format argument.  You should be using %s placeholders in the format,
and then passing each variable as a separate argument.

Here's a version that uses zero forks, and removes the trailing
whitespace correctly.  It's the same parameter expansion with extended
globs that I showed earlier in this thread.

unicorn:~$ cat foo
#!/bin/bash
shopt -s extglob
while read -r line; do
    price=${line##*[[:space:]]}
    retail=$((price * 2))
    product=${line%%+([[:space:]])"$price"}
    printf "The price of '%s' is \$%d.  We will retail it for \$%d.\n" \
	"$product" "$price" "$retail"
done
unicorn:~$ ./foo < data.txt
The price of 'black berry' is $12.  We will retail it for $24.
The price of 'blue berry' is $14.  We will retail it for $28.
The price of 'raspberry' is $9.  We will retail it for $18.
The price of 'huckle berry hound' is $3.  We will retail it for $6.
The price of 'bare-knuckle sandwich' is $27.  We will retail it for $54.

[toc] | [prev] | [next] | [standalone]

#247302

On 4/16/22 09:47, Nicholas Geovanis wrote:

> On 4/15/22 19:37, wilson wrote:
>> Hello
>> 
>> in shell script, how can I use regex to extract values from a 
>> string? maybe value types transformation should be required.
>> 
>> 
>> for instance the string: "black berry 12". I want go get the name: 
>> black berry [String] the price: 12 [Int]
>> 
>> I did this in other language such as java/scala:
>> 
>> scala> val regex = """(.+)\s+(\d+)""".r val regex: 
>> scala.util.matching.Regex = (.+)\s+(\d+)
>> 
>> scala> str match { |   case regex(name,price) => (name,price.toInt)
>> | } val res2: (String, Int) = (black berry,12)
>> 
>> 
>> But sometimes for a simple task I won't want to write a java 
>> program. so what's the shell solution?


When I first read the above, my thought was "Any language can parse 
lines of input.  What are you going to do with $name and $price once you 
have them?"  E.g. think about your input data, your data structures, 
your algorithms, and your output requirements when picking a 
programming/ scripting language.


That said, platform capabilities are also important.  A script written 
to use /bin/sh and "standard system utilities" on Debian can be run on 
any Debian machine without needing to install any packages.  But, the 
same script could very well fail on BSD because GNU userland tools look 
like Unix userland tools, but they are not.  Caveat programmer.


> I didn't want to spoil the party and say this. But now my hand is 
> forced :-) You Mr Original Poster may find perl to be more congenial
> for regular expression use especially. 


+1


> You should find it on every single platform you find bash. 
> Regardless of OS.

I use Bash for an interactive shell because I like its usability 
features.  I use /bin/sh for scripts because of its portability 
(ignoring GNU vs. BSD userland issues).  When I need more power, I use Perl.


But, Perl is no longer as ubiquitous as it used to be.  Perl is not 
installed by default FreeBSD or Cygwin, for example.  Python now seems 
to be the favored super shell scripting language.  I have avoided Python 
because its syntax is whitespace dependent, but someone picking a "P" 
language to learn today should seriously consider Python.


David

[toc] | [prev] | [next] | [standalone]

#247311

On 16/04/2022 14:37, wilson wrote:
> in shell script, how can I use regex to extract values from a string? 
> maybe value types transformation should be required.
> for instance the string: "black berry 12".
> I want go get the name: black berry [String]
> the price: 12 [Int]
> I did this in other language such as java/scala:
> scala> val regex = """(.+)\s+(\d+)""".r
> val regex: scala.util.matching.Regex = (.+)\s+(\d+)
> scala> str match {
>       |   case regex(name,price) => (name,price.toInt)
>       | }
> val res2: (String, Int) = (black berry,12)
> But sometimes for a simple task I won't want to write a java program.
> so what's the shell solution?

Python is a convenient and scriptable solution for many text processing 
problems. You can call it from within bash:


### suppose

X="black berry 12"


### then as a python3 one-liner with -c

python3 -c "import re; m = re.search('(.+)\\s+(\d+)', '$X'); 
print((m.group(1), m.group(2)))"

# prints: ('black berry', '12')


### and you could get the output via command substitution (edit the 
print statement to get just one group; you could do this twice to get 
both items in separate bash variables)

Y=$(python3 -c "import re; m = re.search('(.+)\\s+(\d+)', '$X'); 
print((m.group(1), m.group(2)))")

echo $Y

# prints: ('black berry', '12')


### you can also inline python as a bash here-doc

python3 <<EOF
import re
m = re.search('(.+)\\s+(\d+)', '$X')
print((m.group(1), m.group(2)))
EOF

# prints: ('black berry', '12')


Note the change to your regex: '\' changed to '\\' because bash variable 
interpolation is used to obtain $X . This could be avoided if single 
quotes were used, but then you would need another way to access your 
data, such as reading it from stdin.

It depends on your workflow, but you might consider writing the whole 
thing in Python as a script with first line #!/usr/bin/python3 . Many is 
the time I rewrote a bash script in Python when I realised that I wanted 
proper types, error handling, and unit tests.

Kind regards,

-- 
Ash Joubert <ash@transient.nz>
Director
Transient Software Limited <https://transient.nz/>
New Zealand

[toc] | [prev] | [next] | [standalone]

#247312

On 17/04/2022 12:31, Ash Joubert wrote:
> Note the change to your regex: '\' changed to '\\' because bash variable 
> interpolation is used to obtain $X . This could be avoided if single 
> quotes were used, but then you would need another way to access your 
> data, such as reading it from stdin.

Actually it works fine with just '\' (not '\\') because neither '\s' nor 
'\d' (which I neglected) are bash escape sequences. But in general, 
backlashes might need to be escaped in bash double-quoted strings.

Kind regards,

-- 
Ash Joubert <ash@transient.nz>
Director
Transient Software Limited <https://transient.nz/>
New Zealand

[toc] | [prev] | [next] | [standalone]

#247313

On Sun, Apr 17, 2022 at 12:31:34PM +1200, Ash Joubert wrote:
> Python is a convenient and scriptable solution for many text processing
> problems. You can call it from within bash:

But you really *shouldn't*.  What's the point of writing your script
in two different languages, and having to run two different interpreters?

Just write the whole thing in Python.  (Or Perl, or Tcl, or Ruby, or
whatever language you're using that's not bash.)

[toc] | [prev] | [next] | [standalone]

#247314

On 17/04/2022 12:38, Greg Wooledge wrote:
> On Sun, Apr 17, 2022 at 12:31:34PM +1200, Ash Joubert wrote:
>> Python is a convenient and scriptable solution for many text processing
>> problems. You can call it from within bash:
> But you really *shouldn't*.  What's the point of writing your script
> in two different languages, and having to run two different interpreters?
> Just write the whole thing in Python.  (Or Perl, or Tcl, or Ruby, or
> whatever language you're using that's not bash.)


Indeed! At the end of my email I wrote:


On 17/04/2022 12:31, Ash Joubert wrote:
 > It depends on your workflow, but you might consider writing the whole
 > thing in Python as a script with first line #!/usr/bin/python3 . Many is
 > the time I rewrote a bash script in Python when I realised that I wanted
 > proper types, error handling, and unit tests.


-- 
Ash Joubert <ash@transient.nz>
Director
Transient Software Limited <https://transient.nz/>
New Zealand

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web