Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.php > #14979 > unrolled thread

Run AWK using shell_exec

Back to article view | Back to comp.lang.php

Contents

  Run AWK using shell_exec "Dan'l B" <dan.hinckley@gmail.com> - 2015-02-26 14:40 -0800
    Re: Run AWK using shell_exec Jerry Stuckle <jstucklex@attglobal.net> - 2015-02-26 19:50 -0500
      Re: Run AWK using shell_exec "Dan'l B" <dan.hinckley@gmail.com> - 2015-02-26 17:21 -0800
      Re: Run AWK using shell_exec "Dan'l B" <dan.hinckley@gmail.com> - 2015-02-26 17:41 -0800
        Re: Run AWK using shell_exec Jerry Stuckle <jstucklex@attglobal.net> - 2015-02-26 20:51 -0500
        Re: Run AWK using shell_exec Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-02-27 02:23 +0000
          Re: Run AWK using shell_exec "Dan'l B" <dan.hinckley@gmail.com> - 2015-02-26 19:11 -0800
            Re: Run AWK using shell_exec Matthew Carter <m@ahungry.com> - 2015-02-27 00:28 -0500
            Re: Run AWK using shell_exec Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-02-27 11:14 +0000
          Re: Run AWK using shell_exec Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-02-27 14:17 +0100
            Re: Run AWK using shell_exec "Dan'l B" <dan.hinckley@gmail.com> - 2015-02-27 11:40 -0800
              Re: Run AWK using shell_exec Jerry Stuckle <jstucklex@attglobal.net> - 2015-02-27 16:03 -0500

#14979 — Run AWK using shell_exec

Given a file with data such as

2015-12-24 22:02 12   9.87 feet  High Tide
2015-12-25 03:33 12  -0.38 feet  Low Tide
2015-12-25 06:11 12   Full Moon
2015-12-25 10:16 12  11.01 feet  High Tide
2015-12-25 16:09 12  -1.29 feet  Low Tide

This awk command works to get the minimum value in col (excluding text):

awk 'min=="" || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min}' fullyear.txt

but I am stuck about how to format it for shell_exec.

For example, with full error reporting on, this seems to fail:

$maxLow = shell_exec("/usr/bin/awk 'min=='' || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min} fullyear.txt");

[toc] | [next] | [standalone]

#14980

On 2/26/2015 5:40 PM, Dan'l B wrote:
> Given a file with data such as
> 
> 2015-12-24 22:02 12   9.87 feet  High Tide
> 2015-12-25 03:33 12  -0.38 feet  Low Tide
> 2015-12-25 06:11 12   Full Moon
> 2015-12-25 10:16 12  11.01 feet  High Tide
> 2015-12-25 16:09 12  -1.29 feet  Low Tide
> 
> This awk command works to get the minimum value in col (excluding text):
> 
> awk 'min=="" || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min}' fullyear.txt
> 
> but I am stuck about how to format it for shell_exec.
> 
> For example, with full error reporting on, this seems to fail:
> 
> $maxLow = shell_exec("/usr/bin/awk 'min=='' || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min} fullyear.txt");
> 

I'm not that familiar with awk syntax, but I can see why it doesn't work
- you're not passing the same parameters to awk from PHP as you use in
the command line.

Not tested, but something like this might work:

$maxLow = shell_exec(\''min=="" || $4~/^[^[:alpha:]]+$/ && $4 < min
{min=$4} END{print min}\' fullyear.txt');

Or

$maxLow = shell_exec("'min==\"\" || $4~/^[^[:alpha:]]+$/ && $4 < min
{min=$4} END{print min}' fullyear.txt");

-- 
==================
Remove the "x" from my email address
Jerry Stuckle
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]

#14981

On Thursday, February 26, 2015 at 7:50:32 PM UTC-5, Jerry Stuckle wrote:
> On 2/26/2015 5:40 PM, Dan'l B wrote:
> > Given a file with data such as
> > 
> > 2015-12-24 22:02 12   9.87 feet  High Tide
> > 2015-12-25 03:33 12  -0.38 feet  Low Tide
> > 2015-12-25 06:11 12   Full Moon
> > 2015-12-25 10:16 12  11.01 feet  High Tide
> > 2015-12-25 16:09 12  -1.29 feet  Low Tide
> > 
> > This awk command works to get the minimum value in col (excluding text):
> > 
> > awk 'min=="" || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min}' fullyear.txt
> > 
> > but I am stuck about how to format it for shell_exec.
> > 
> > For example, with full error reporting on, this seems to fail:
> > 
> > $maxLow = shell_exec("/usr/bin/awk 'min=='' || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min} fullyear.txt");
> > 
> 
> I'm not that familiar with awk syntax, but I can see why it doesn't work
> - you're not passing the same parameters to awk from PHP as you use in
> the command line.
> 
> Not tested, but something like this might work:
> 
> $maxLow = shell_exec(\''min=="" || $4~/^[^[:alpha:]]+$/ && $4 < min
> {min=$4} END{print min}\' fullyear.txt');
> 
> Or
> 
> $maxLow = shell_exec("'min==\"\" || $4~/^[^[:alpha:]]+$/ && $4 < min
> {min=$4} END{print min}' fullyear.txt");
> 
> -- 
> ==================
> Remove the "x" from my email address
> Jerry Stuckle

> ==================

You got me on the right track; this works, as you had to add back the awk command with its path:

$maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min}' fullyear.txt"); 

Many thanks!

[toc] | [prev] | [next] | [standalone]

#14982

On Thursday, February 26, 2015 at 7:50:32 PM UTC-5, Jerry Stuckle wrote:
> On 2/26/2015 5:40 PM, Dan'l B wrote:
> > Given a file with data such as
> > 
> > 2015-12-24 22:02 12   9.87 feet  High Tide
> > 2015-12-25 03:33 12  -0.38 feet  Low Tide
> > 2015-12-25 06:11 12   Full Moon
> > 2015-12-25 10:16 12  11.01 feet  High Tide
> > 2015-12-25 16:09 12  -1.29 feet  Low Tide
> > 
> > This awk command works to get the minimum value in col (excluding text):
> > 
> > awk 'min=="" || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min}' fullyear.txt
> > 
> > but I am stuck about how to format it for shell_exec.
> > 
> > For example, with full error reporting on, this seems to fail:
> > 
> > $maxLow = shell_exec("/usr/bin/awk 'min=='' || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min} fullyear.txt");
> > 
> 
> I'm not that familiar with awk syntax, but I can see why it doesn't work
> - you're not passing the same parameters to awk from PHP as you use in
> the command line.
> 
> Not tested, but something like this might work:
> 
> $maxLow = shell_exec(\''min=="" || $4~/^[^[:alpha:]]+$/ && $4 < min
> {min=$4} END{print min}\' fullyear.txt');
> 
> Or
> 
> $maxLow = shell_exec("'min==\"\" || $4~/^[^[:alpha:]]+$/ && $4 < min
> {min=$4} END{print min}' fullyear.txt");
> 
> -- 
> ==================
> Remove the "x" from my email address
> Jerry Stuckle
>
> ==================

Ooops, one more issue:

Substituting a variable containing a path-to-the file, I can't seem to get the escaping right:

Works:
$maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min}' fullyear.txt"); 

Fails: (where $tidesFilePath is an absolute path to the fullyear.txt file).
$tidesFilePath = "/Tidefiles/fullyear.txt"; (example path)
$maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min}'" . $tidesFilePath . \""); 

[toc] | [prev] | [next] | [standalone]

#14983

On 2/26/2015 8:41 PM, Dan'l B wrote:
> On Thursday, February 26, 2015 at 7:50:32 PM UTC-5, Jerry Stuckle wrote:
>> On 2/26/2015 5:40 PM, Dan'l B wrote:
>>> Given a file with data such as
>>>
>>> 2015-12-24 22:02 12   9.87 feet  High Tide
>>> 2015-12-25 03:33 12  -0.38 feet  Low Tide
>>> 2015-12-25 06:11 12   Full Moon
>>> 2015-12-25 10:16 12  11.01 feet  High Tide
>>> 2015-12-25 16:09 12  -1.29 feet  Low Tide
>>>
>>> This awk command works to get the minimum value in col (excluding text):
>>>
>>> awk 'min=="" || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min}' fullyear.txt
>>>
>>> but I am stuck about how to format it for shell_exec.
>>>
>>> For example, with full error reporting on, this seems to fail:
>>>
>>> $maxLow = shell_exec("/usr/bin/awk 'min=='' || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min} fullyear.txt");
>>>
>>
>> I'm not that familiar with awk syntax, but I can see why it doesn't work
>> - you're not passing the same parameters to awk from PHP as you use in
>> the command line.
>>
>> Not tested, but something like this might work:
>>
>> $maxLow = shell_exec(\''min=="" || $4~/^[^[:alpha:]]+$/ && $4 < min
>> {min=$4} END{print min}\' fullyear.txt');
>>
>> Or
>>
>> $maxLow = shell_exec("'min==\"\" || $4~/^[^[:alpha:]]+$/ && $4 < min
>> {min=$4} END{print min}' fullyear.txt");
>>
>> -- 
>> ==================
>> Remove the "x" from my email address
>> Jerry Stuckle
>>
>> ==================
> 
> Ooops, one more issue:
> 
> Substituting a variable containing a path-to-the file, I can't seem to get the escaping right:
> 
> Works:
> $maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min}' fullyear.txt"); 
> 
> Fails: (where $tidesFilePath is an absolute path to the fullyear.txt file).
> $tidesFilePath = "/Tidefiles/fullyear.txt"; (example path)
> $maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/ && $4 < min {min=$4} END{print min}'" . $tidesFilePath . \""); 
> 

I suspect your problem is the absolute path is incorrect.  What's
actually in $tidesFilePath at execution time - and does the path
actually exist?

Sometimes it helps to set a string to your command and print the string
before trying to execute the command, i.e.

$cmd = "/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/ && $4 < min
{min=$4} END{print min}'" . $tidesFilePath . \"";
echo $cmd . "\n";
shell_exec($cmd);

Does this help?

-- 
==================
Remove the "x" from my email address
Jerry Stuckle
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]

#14984

"Dan'l B" <dan.hinckley@gmail.com> writes:
<snip>
> Substituting a variable containing a path-to-the file, I can't seem to
> get the escaping right:
>
> Works:
> $maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/
> && $4 < min {min=$4} END{print min}' fullyear.txt");
>
> Fails: (where $tidesFilePath is an absolute path to the fullyear.txt file).
> $tidesFilePath = "/Tidefiles/fullyear.txt"; (example path)
> $maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/
> && $4 < min {min=$4} END{print min}'" . $tidesFilePath . \"");

That's not syntactically correct PHP.  I'm not sure what the . \"" bit
at the end is supposed to do, but you don't need it.  You *do* need a
space between the end of the awk script in single quotes and the file
name:

 ... $4 < min {min=$4} END{print min}' " . $tidesFilePath);

But you should be very careful when including variables into a shell
command.  Never forget little Bobby Tables http://www.xkcd.com/327/

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#14985

On Thursday, February 26, 2015 at 9:23:19 PM UTC-5, Ben Bacarisse wrote:
> "Dan'l B" <> writes:
> <snip>
> > Substituting a variable containing a path-to-the file, I can't seem to
> > get the escaping right:
> >
> > Works:
> > $maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/
> > && $4 < min {min=$4} END{print min}' fullyear.txt");
> >
> > Fails: (where $tidesFilePath is an absolute path to the fullyear.txt file).
> > $tidesFilePath = "/Tidefiles/fullyear.txt"; (example path)
> > $maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/
> > && $4 < min {min=$4} END{print min}'" . $tidesFilePath . \"");
> 
> That's not syntactically correct PHP.  I'm not sure what the . \"" bit
> at the end is supposed to do, but you don't need it.  You *do* need a
> space between the end of the awk script in single quotes and the file
> name:
> 
>  ... $4 < min {min=$4} END{print min}' " . $tidesFilePath);
> 
> But you should be very careful when including variables into a shell
> command.  Never forget little Bobby Tables http://www.xkcd.com/327/
> 
Thanks, still doesn't seem to work but I'll double-check the path. I do grasp the problem of variables in shell commands but is there an alternative way to get the highest and lowest values in a column of text in a multi-column text file using only PHP?

[toc] | [prev] | [next] | [standalone]

#14986

"Dan'l B" <dan.hinckley@gmail.com> writes:

> On Thursday, February 26, 2015 at 9:23:19 PM UTC-5, Ben Bacarisse wrote:
>> "Dan'l B" <> writes:
>> <snip>
>> > Substituting a variable containing a path-to-the file, I can't seem to
>> > get the escaping right:
>> >
>> > Works:
>> > $maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/
>> > && $4 < min {min=$4} END{print min}' fullyear.txt");
>> >
>> > Fails: (where $tidesFilePath is an absolute path to the fullyear.txt file).
>> > $tidesFilePath = "/Tidefiles/fullyear.txt"; (example path)
>> > $maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/
>> > && $4 < min {min=$4} END{print min}'" . $tidesFilePath . \"");
>> 
>> That's not syntactically correct PHP.  I'm not sure what the . \"" bit
>> at the end is supposed to do, but you don't need it.  You *do* need a
>> space between the end of the awk script in single quotes and the file
>> name:
>> 
>>  ... $4 < min {min=$4} END{print min}' " . $tidesFilePath);
>> 
>> But you should be very careful when including variables into a shell
>> command.  Never forget little Bobby Tables http://www.xkcd.com/327/
>> 
> Thanks, still doesn't seem to work but I'll double-check the path. I
> do grasp the problem of variables in shell commands but is there an
> alternative way to get the highest and lowest values in a column of
> text in a multi-column text file using only PHP?

This works on your sample data:

<?php

$data = array_filter (array_map (
  function ($l)
  {
    return explode (' ', preg_replace ('/\s+/', ' ', $l))[3];
  }, explode ("\n", file_get_contents ('/tmp/hi-low'))), 'is_numeric');

sort ($data); 

$d1 = $d2 = $data; // This is done incase there is only one entry

$low = array_shift ($d1);
$hi  = array_pop ($d2);

var_dump ($low, $hi);

Fine tune as needed of course (this assumes one or more whitespace such
as tab or space breaks the 'columns').  You'd probably want your data to
stick to a real column format, such as CSV or TSV though.

-- 
Matthew Carter (m@ahungry.com)
http://ahungry.com

[toc] | [prev] | [next] | [standalone]

#14987

"Dan'l B" <dan.hinckley@gmail.com> writes:
<snip>
> Thanks, still doesn't seem to work but I'll double-check the path. I
> do grasp the problem of variables in shell commands but is there an
> alternative way to get the highest and lowest values in a column of
> text in a multi-column text file using only PHP?

There's been one answer, but here's a different one just for fun:

  if (preg_match_all('/^(?:[^\s]+\s+){3}(-?[\d.]+)/m',
                   file_get_contents('fullyear.txt'),
                   $matches))
      echo min($matches[1]);

(The number pattern is a little generous, but that may not matter.)

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#14988

Ben Bacarisse wrote:

> "Dan'l B" <dan.hinckley@gmail.com> writes:
>> Fails: (where $tidesFilePath is an absolute path to the fullyear.txt
>> file). $tidesFilePath = "/Tidefiles/fullyear.txt"; (example path)
>> $maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/
>> && $4 < min {min=$4} END{print min}'" . $tidesFilePath . \"");
> 
> That's not syntactically correct PHP.  I'm not sure what the . \"" bit
> at the end is supposed to do, but you don't need it.  You *do* need a
> space between the end of the awk script in single quotes and the file
> name:
> 
>  ... $4 < min {min=$4} END{print min}' " . $tidesFilePath);
> 
> But you should be very careful when including variables into a shell
> command.  Never forget little Bobby Tables http://www.xkcd.com/327/

Rule of thumb: Whenever you use any of the shell-accessing features, use 
escapeshellcmd() and escapeshellarg() as well.

Whenever you find yourself using PHP’s shell-accessing features to run an 
ad-hoc AWK script from PHP, you are doing something fundamentally wrong.  
PHP can do everything that AWK can do, and more (for example, it supports 
PCRE which even GNU awk does not), without requiring for that the additional 
privileges (and opening potential security leaks) that are required to run 
other programs from PHP.  This applies to ad-hoc sed(1) and perl(1) scripts 
as well.

<http://php.net/escapeshellcmd>
<http://php.net/file_get_contents>
<http://php.net/preg_replace>
 
-- 
PointedEars
Zend Certified PHP Engineer
Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.

[toc] | [prev] | [next] | [standalone]

#14989

On Friday, February 27, 2015 at 8:18:49 AM UTC-5, Thomas 'PointedEars' Lahn wrote:
> Ben Bacarisse wrote:
> 
> > "Dan'l B"  writes:
> >> Fails: (where $tidesFilePath is an absolute path to the fullyear.txt
> >> file). $tidesFilePath = "/Tidefiles/fullyear.txt"; (example path)
> >> $maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/
> >> && $4 < min {min=$4} END{print min}'" . $tidesFilePath . \"");
> > 
> > That's not syntactically correct PHP.  I'm not sure what the . \"" bit
> > at the end is supposed to do, but you don't need it.  You *do* need a
> > space between the end of the awk script in single quotes and the file
> > name:
> > 
> >  ... $4 < min {min=$4} END{print min}' " . $tidesFilePath);
> > 
> > But you should be very careful when including variables into a shell
> > command.  Never forget little Bobby Tables http://www.xkcd.com/327/
> 
> Rule of thumb: Whenever you use any of the shell-accessing features, use 
> escapeshellcmd() and escapeshellarg() as well.
> 
> Whenever you find yourself using PHP's shell-accessing features to run an 
> ad-hoc AWK script from PHP, you are doing something fundamentally wrong.  
> PHP can do everything that AWK can do, and more (for example, it supports 
> PCRE which even GNU awk does not), without requiring for that the additional 
> privileges (and opening potential security leaks) that are required to run 
> other programs from PHP.  This applies to ad-hoc sed(1) and perl(1) scripts 
> as well.
> 
> <http://php.net/escapeshellcmd>
> <http://php.net/file_get_contents>
> <http://php.net/preg_replace>
>  
> -- 
> PointedEars
> Zend Certified PHP Engineer
> Twitter: @PointedEars2
> Please do not cc me. / Bitte keine Kopien per E-Mail.

Thanks to one and all for the enlightening examples, pointers and lectures; very valuable for a noob like me.

[toc] | [prev] | [next] | [standalone]

#14990

On 2/27/2015 2:40 PM, Dan'l B wrote:
> On Friday, February 27, 2015 at 8:18:49 AM UTC-5, Thomas 'PointedEars' Lahn wrote:
>> Ben Bacarisse wrote:
>>
>>> "Dan'l B"  writes:
>>>> Fails: (where $tidesFilePath is an absolute path to the fullyear.txt
>>>> file). $tidesFilePath = "/Tidefiles/fullyear.txt"; (example path)
>>>> $maxLow = shell_exec("/usr/bin/awk 'min==\"\" || $4~/^[^[:alpha:]]+$/
>>>> && $4 < min {min=$4} END{print min}'" . $tidesFilePath . \"");
>>>
>>> That's not syntactically correct PHP.  I'm not sure what the . \"" bit
>>> at the end is supposed to do, but you don't need it.  You *do* need a
>>> space between the end of the awk script in single quotes and the file
>>> name:
>>>
>>>  ... $4 < min {min=$4} END{print min}' " . $tidesFilePath);
>>>
>>> But you should be very careful when including variables into a shell
>>> command.  Never forget little Bobby Tables http://www.xkcd.com/327/
>>
>> Rule of thumb: Whenever you use any of the shell-accessing features, use 
>> escapeshellcmd() and escapeshellarg() as well.
>>
>> Whenever you find yourself using PHP's shell-accessing features to run an 
>> ad-hoc AWK script from PHP, you are doing something fundamentally wrong.  
>> PHP can do everything that AWK can do, and more (for example, it supports 
>> PCRE which even GNU awk does not), without requiring for that the additional 
>> privileges (and opening potential security leaks) that are required to run 
>> other programs from PHP.  This applies to ad-hoc sed(1) and perl(1) scripts 
>> as well.
>>
>> <http://php.net/escapeshellcmd>
>> <http://php.net/file_get_contents>
>> <http://php.net/preg_replace>
>>  
>> -- 
>> PointedEars
>> Zend Certified PHP Engineer
>> Twitter: @PointedEars2
>> Please do not cc me. / Bitte keine Kopien per E-Mail.
> 
> Thanks to one and all for the enlightening examples, pointers and lectures; very valuable for a noob like me.
> 

Dan'l,

Don't worry too much about what "Pointed Head" says.  He's a resident
troll; anyone who doesn't meet his specifications is "fundamentally wrong".

Using awk isn't necessarily the best way to do it, for a number of
reasons (i.e. many hosting companies have shell_exec() disabled due to
security reasons).  And yes, you *can* do most anything in PHP that you
can do in awk.

But that doesn't mean that awk is "fundamentally wrong".  It's just
another way of doing things.  And a new PHP programmer like yourself has
a lot of things to learn; I don't see using something you're already
familiar with (and have working in another environment) as being
"wrong".  You can always go back and change it later once you've learned
more PHP.


-- 
==================
Remove the "x" from my email address
Jerry Stuckle
jstucklex@attglobal.net
==================

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.php

csiph-web