Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #36272

Re: Need a specific sort of string modification. Can someone help?

Show all headers | View raw

In article <roy-103D43.15470305012013@news.panix.com>,
 Roy Smith <roy@panix.com> wrote:

> It's rare to find applications these days that are truly CPU bound.  
> Once you've used some reasonable algorithm, i.e. not done anything in 
> O(n^2) that could have been done in O(n) or O(n log n), you will more 
> often run up against I/O speed, database speed, network latency, memory 
> exhaustion, or some such as the reason your code is too slow.

Well, I just found a counter-example :-)

I've been doing some log analysis.  It's been taking a grovelingly long 
time, so I decided to fire up the profiler and see what's taking so 
long.  I had a pretty good idea of where the ONLY TWO POSSIBLE hotspots 
might be (looking up IP addresses in the geolocation database, or 
producing some pretty pictures using matplotlib).  It was just a matter 
of figuring out which it was.

As with most attempts to out-guess the profiler, I was totally, 
absolutely, and embarrassingly wrong.

It turns out we were spending most of the time parsing timestamps!  
Since there's no convenient way (I don't consider strptime() to be 
convenient) to parse isoformat strings in the standard library, our 
habit has been to use the oh-so-simple parser from the third-party 
dateutil package.  Well, it turns out that's slow as all get-out 
(probably because it's trying to be smart about auto-recognizing 
formats).  For the test I ran (on a few percent of the real data), we 
spent 90 seconds in parse().

OK, so I dragged out the strptime() docs and built the stupid format 
string (%Y-%m-%dT%H:%M:%S+00:00).  That got us down to 25 seconds in 
strptime().

But, I could also see it was spending a significant amount in routines 
that looked like they were computing things like day of the week that we 
didn't need.  For what I was doing, we only really needed the hour and 
minute.  So I tried:

        t_hour = int(date[11:13])
        t_minute = int(date[14:16])

that got us down to 12 seconds overall (including the geolocation and 
pretty pictures).

I think it turns out we never do anything with the hour and minute other 
than print them back out, so just

       t_hour_minute = date[11:16]

would probably be good enough, but I think I'm going to stop where I am 
and declare victory :-)

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Need a specific sort of string modification. Can someone help? Sia <hossein.asgharian@gmail.com> - 2013-01-05 00:35 -0800
  Re: Need a specific sort of string modification. Can someone help? Frank Millman <frank@chagford.com> - 2013-01-05 11:15 +0200
  Re: Need a specific sort of string modification. Can someone help? Chris Angelico <rosuav@gmail.com> - 2013-01-05 20:27 +1100
    Re: Need a specific sort of string modification. Can someone help? Roy Smith <roy@panix.com> - 2013-01-05 09:30 -0500
      Re: Need a specific sort of string modification. Can someone help? Chris Angelico <rosuav@gmail.com> - 2013-01-06 01:47 +1100
        Re: Need a specific sort of string modification. Can someone help? Roy Smith <roy@panix.com> - 2013-01-05 10:03 -0500
          Re: Need a specific sort of string modification. Can someone help? Chris Angelico <rosuav@gmail.com> - 2013-01-06 02:09 +1100
            Re: Need a specific sort of string modification. Can someone help? Roy Smith <roy@panix.com> - 2013-01-05 10:38 -0500
              Re: Need a specific sort of string modification. Can someone help? Chris Angelico <rosuav@gmail.com> - 2013-01-06 02:57 +1100
              Re: Need a specific sort of string modification. Can someone help? Ian Kelly <ian.g.kelly@gmail.com> - 2013-01-05 13:04 -0700
              Re: Need a specific sort of string modification. Can someone help? Chris Angelico <rosuav@gmail.com> - 2013-01-06 07:32 +1100
                Re: Need a specific sort of string modification. Can someone help? Roy Smith <roy@panix.com> - 2013-01-05 15:47 -0500
                Re: Need a specific sort of string modification. Can someone help? Roy Smith <roy@panix.com> - 2013-01-06 12:28 -0500
                Re: Need a specific sort of string modification. Can someone help? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-06 23:19 +0000
  Re: Need a specific sort of string modification. Can someone help? Roy Smith <roy@panix.com> - 2013-01-05 09:12 -0500
  Re: Need a specific sort of string modification. Can someone help? Tim Chase <python.list@tim.thechases.com> - 2013-01-05 11:24 -0600
  Re: Need a specific sort of string modification. Can someone help? Tim Chase <python.list@tim.thechases.com> - 2013-01-05 12:49 -0600
  Re: Need a specific sort of string modification. Can someone help? Mitya Sirenef <msirenef@lightbird.net> - 2013-01-06 01:32 -0500
  Re: Need a specific sort of string modification. Can someone help? Mitya Sirenef <msirenef@lightbird.net> - 2013-01-06 14:53 -0500
  Re: Need a specific sort of string modification. Can someone help? Nick Mellor <thebalancepro@gmail.com> - 2013-01-06 18:48 -0800
  Re: Need a specific sort of string modification. Can someone help? Nick Mellor <thebalancepro@gmail.com> - 2013-01-06 19:40 -0800
    Re: Need a specific sort of string modification. Can someone help? Nick Mellor <thebalancepro@gmail.com> - 2013-01-06 21:28 -0800
  Re: Need a specific sort of string modification. Can someone help? Nick Mellor <thebalancepro@gmail.com> - 2013-01-06 21:30 -0800
  Re: Need a specific sort of string modification. Can someone help? John Ladasky <john_ladasky@sbcglobal.net> - 2013-01-06 21:39 -0800
  Re: Need a specific sort of string modification. Can someone help? Nick Mellor <thebalancepro@gmail.com> - 2013-01-06 23:07 -0800

csiph-web