Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #67953 > unrolled thread

Re: script uses up all memory

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: script uses up all memory Larry Martell <larry.martell@gmail.com> - 2014-03-06 16:56 -0500

#67953 — Re: script uses up all memory

On Wed, Mar 5, 2014 at 5:27 PM, Larry Martell <larry.martell@gmail.com> wrote:
> I have a script that forks off other processes and attempts to manage
> them. Here is a stripped down version of the script:
>
>         self.sleepTime = 300
>         self.procs = {}
>         self.startTimes = {}
>         self.cmd = ['python', '/usr/local/motor/motor/app/some_other_script.py']
>
>         while True:
>             try:
>                 self.tools = Tool.objects.filter(ip__isnull=False)
>             except Exception, e:
>                 print 'error from django call: ' + str(e)
>                 sys.exit(1)
>
>             for tool in self.tools:
>                 name = tool.name
>                 if name in self.procs:
>                     if self.procs[name].poll() is None:
>                         if
> (datetime.datetime.now()-self.startTimes[name]) >
> datetime.timedelta(hours=12):
>                             # it's been running too long - kill it
>                             print 'killing script for ' + name + "
> it's been running too long"
>                             self.procs[name].kill()
>                         else:
>                             continue
>
>                     if self.procs[name].returncode:
>                         print 'scrikpt failed for ' + name + ', error
> = ' + str(self.procs[name].returncode)
>
>                     print 'starting script.py for ' + name + ' at ' +
> str(datetime.datetime.now())
>                     try:
>                         self.procs[name] = subprocess.Popen(self.cmd)
>                         self.startTimes[name] = datetime.datetime.now()
>                     except Exception, e:
>                         print 'error from Popen: ' + str(e)
>                         sys.exit(1)
>                 else:
>                     print 'starting script.py for ' + name + ' at ' +
> str(datetime.datetime.now())
>                     try:
>                         self.procs[name] = subprocess.Popen(self.cmd)
>                         self.startTimes[name] = datetime.datetime.now()
>                     except Exception, e:
>                         print 'error from Popen: ' + str(e)
>                         sys.exit(1)
>
>                 time.sleep(self.sleepTime)
>
>
> The script does what it's intended to do, however after about 2 hours
> it has used up all the memory available and the machine hangs. Can
> anyone see something that I am doing here that would be using memory
> like this? Perhaps some immutable object needs to be repeatedly
> recreated?

I figured out what is causing this. Each pass through the loop it does:

self.tools = Tool.objects.filter(ip__isnull=False)

And that is what is causing the memory consumption. If I move that
outside the loop and just do that once the memory issue goes away. Now
I need to figure out why this is happening and how to prevent it as
they do want to query the db each pass through the loop in case it has
been updated.

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web