Re: Convert AWK regex to Python

Path	csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!feeder.news-service.com!kanaga.switch.ch!switch.ch!newsserver.news.garr.it!newsserver.cilea.it!not-for-mail
From	Giacomo Boffi <giacomo.boffi@polimi.it>
Newsgroups	comp.lang.python
Subject	Re: Convert AWK regex to Python
Date	Mon, 16 May 2011 16:19:49 +0200
Organization	The Sun and The Rain
Lines	49
Message-ID	<86iptabud6.fsf@aiuole.stru.polimi.it> (permalink)
References	<d4b4303e-8002-4f5f-9cb8-af7ab7451f13@glegroupsg2000goo.googlegroups.com>
NNTP-Posting-Host	aiuole.stru.polimi.it
Mime-Version	1.0
Content-Type	text/plain; charset=us-ascii
X-Trace	newsserver.cilea.it 1305555447 29938 131.175.189.141 (16 May 2011 14:17:27 GMT)
X-Complaints-To	news@cilea.it
NNTP-Posting-Date	16 May 2011 14:17:27 GMT
User-Agent	Gnus/5.1008 (Gnus v5.10.8) XEmacs/21.5-b31 (linux)
Cancel-Lock	sha1:P9CTv+mhzAT7MR42TzDC+LCK5Pg=
Xref	x330-a1.tempe.blueboxinc.net comp.lang.python:5519

Show key headers only | View raw

J <jnr.gonzalez@googlemail.com> writes:

> cat logs/pdu_log_fe.log | awk -F\- '{print $1,$NF}' | awk -F\. '{print $1,$NF}' | awk '{print $1,$4,$5}' | sort | uniq | while read service command status; do echo "Service: $service, Command: $command, Status: $status, Occurrences: `grep $service logs/pdu_log_fe.log | grep $command | grep $status | wc -l | awk '{ print $1 }'`" >> logs/pdu_log_fe_clean.log; done
>
> This AWK command gets lines which look like this:-
>
> 2011-05-16 09:46:22,361 [Thread-4847133] PDU D <G_CC_SMS_SERVICE_51408_656.O_ CC_SMS_SERVICE_51408_656-ServerThread-VASPSessionThread-7ee35fb0-7e87-11e0-a2da-00238bce423b-TRX - 2011-05-16 09:46:22 - OUT - (submit_resp: (pdu: L: 53 ID: 80000004 Status: 0 SN: 25866) 98053090-7f90-11e0-a2da-00238bce423b (opt: ) ) >
>
> And outputs lines like this:-
>
> CC_SMS_SERVICE_51408 submit_resp: 0
>

i see some discrepancies in the description of your problem

1. if i echo a properly quoted line "like this" above in the pipeline
   formed by the first three awk commands i get

$ echo $likethis | awk -F\- '{print $1,$NF}' \
                 | awk -F\. '{print$1,$NF}'  \
                 | awk '{print $1,$4,$5}'
2011 ) )
$ 
   not a triple 'service command status'

2. with regard to the final product, you script outputs lines like in

echo "Service: $service, [...]"

   and you say that it produces lines like

CC_SMS_SERVICE_51408 submit_resp: 


WHATEVER, the abnormous run time is due to the fact that for every
output line you rescan again and again the whole log file

IF i had understood what you want, imho you should run your data
through sort and uniq -c

$ awk -F\- '{print $1,$NF}' < $file \
        | awk -F\. '{print$1,$NF}'  \
        | awk '{print $1,$4,$5}'    | sort | uniq -c | format_program

uniq -c drops repeated lines from a sorted input AND prepends to each
line the count of equal lines in the original stream

hth
                                                                g

Thread

Convert AWK regex to Python J <jnr.gonzalez@googlemail.com> - 2011-05-16 01:19 -0700
  Re: Convert AWK regex to Python Chris Angelico <rosuav@gmail.com> - 2011-05-16 18:31 +1000
  Re: Convert AWK regex to Python Peter Otten <__peter__@web.de> - 2011-05-16 12:07 +0200
  Re: Convert AWK regex to Python Giacomo Boffi <giacomo.boffi@polimi.it> - 2011-05-16 16:19 +0200
    Re: Convert AWK regex to Python Matt Berends <matthewberends@gmail.com> - 2011-05-16 08:02 -0700
  Re: Convert AWK regex to Python MRAB <python@mrabarnett.plus.com> - 2011-05-16 17:39 +0100

csiph-web