Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #25979 > unrolled thread

Daemon loses __file__ reference after a while.

Back to article view | Back to comp.lang.python

Contents

  Daemon loses __file__ reference after a while. "ivdneut@gmail.com" <ivdneut@gmail.com> - 2012-07-24 04:48 -0700
    Re: Daemon loses __file__ reference after a while. Ervin Hegedüs <airween@gmail.com> - 2012-07-24 14:24 +0200
      Re: Daemon loses __file__ reference after a while. "ivdneut@gmail.com" <ivdneut@gmail.com> - 2012-07-24 05:41 -0700
      Re: Daemon loses __file__ reference after a while. "ivdneut@gmail.com" <ivdneut@gmail.com> - 2012-07-24 05:41 -0700
    Re: Daemon loses __file__ reference after a while. Laszlo Nagy <gandalf@shopzeus.com> - 2012-07-24 14:31 +0200
    Re: Daemon loses __file__ reference after a while. Dieter Maurer <dieter@handshake.de> - 2012-07-24 16:59 +0200
      Re: Daemon loses __file__ reference after a while. Paul Rubin <no.email@nospam.invalid> - 2012-07-24 12:32 -0700
        Re: Daemon loses __file__ reference after a while. Ian Kelly <ian.g.kelly@gmail.com> - 2012-07-24 14:08 -0600
    Re: Daemon loses __file__ reference after a while. Ian Kelly <ian.g.kelly@gmail.com> - 2012-07-24 14:02 -0600

#25979 — Daemon loses __file__ reference after a while.

Hello,

I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:

Exception info: Traceback (most recent call last):
  File "scheduler.py", line 376, in applyrule
    result = execrule(rule_code)
  File "scheduler.py", line 521, in execrule
    rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
NameError: name '__file__' is not defined

This section of the code is executed in this process *all the time*, but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.

I am running python from a virtual-env installation from a stock Red Hat EL 6.2 installation:

(virtual-env)[user@host ~]$ python --version
Python 2.6.6
(virtual-env)[user@host ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.2 (Santiago)

I would greatly appreciate any pointers on where to start looking to find the problem.

Ian.

[toc] | [next] | [standalone]

#25982

hello,

On Tue, Jul 24, 2012 at 04:48:42AM -0700, ivdneut@gmail.com wrote:
> Hello,
> 
> I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
> 
> Exception info: Traceback (most recent call last):
>   File "scheduler.py", line 376, in applyrule
>     result = execrule(rule_code)
>   File "scheduler.py", line 521, in execrule
>     rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
> NameError: name '__file__' is not defined
> 
> This section of the code is executed in this process *all the time*, but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.

could you send the relevant part of the code?

I mean: how do you daemonize your process?
 
> I am running python from a virtual-env installation from a stock Red Hat EL 6.2 installation:
> 
> (virtual-env)[user@host ~]$ python --version
> Python 2.6.6
> (virtual-env)[user@host ~]$ cat /etc/redhat-release 
> Red Hat Enterprise Linux Server release 6.2 (Santiago)

If you use fork(), it drops all file descriptors, and creates new
ones - may be then loss the __file__...?


a.


-- 
I � UTF-8

[toc] | [prev] | [next] | [standalone]

#25985

On Tuesday, July 24, 2012 2:24:31 PM UTC+2, Ervin Hegedüs wrote:
> hello,
> 
> On Tue, Jul 24, 2012 at 04:48:42AM -0700, ivdneut@gmail.com wrote:
> > Hello,
> > 
> > I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
> > 
> > Exception info: Traceback (most recent call last):
> >   File "scheduler.py", line 376, in applyrule
> >     result = execrule(rule_code)
> >   File "scheduler.py", line 521, in execrule
> >     rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
> > NameError: name '__file__' is not defined
> > 
> > This section of the code is executed in this process *all the time*, but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.
> 
> could you send the relevant part of the code?
> 
> I mean: how do you daemonize your process?

It's done by a double fork:

## First fork()
pid = os.fork()
if pid != 0: sys.exit(0) # parent exits.

## create new session
os.setsid()
	   
## ignore SIGHUP
signal.signal(signal.SIGHUP, signal.SIG_IGN)

## Second fork()
pid = os.fork()
if pid != 0: sys.exit(0) # First child exits.
	   
## Change working directory to the home directory.
homedir = pwd.getpwuid(os.geteuid())[5]
os.chdir(homedir)

os.umask(0)

for fd in range(0, 1024):
try:
    os.close(fd)
except:
    pass # fd not open, ignore this exception.

The original C version of this code is from W.R. Stevens' daemon_init() routine in "UNIX Network Programming Volume 1, second edition"

>  
> > I am running python from a virtual-env installation from a stock Red Hat EL 6.2 installation:
> > 
> > (virtual-env)[user@host ~]$ python --version
> > Python 2.6.6
> > (virtual-env)[user@host ~]$ cat /etc/redhat-release 
> > Red Hat Enterprise Linux Server release 6.2 (Santiago)
> 
> If you use fork(), it drops all file descriptors, and creates new
> ones - may be then loss the __file__...?

I doubt this would be it, or it would stop working immediately, since daemonization is done upon startup of the process. File descriptors are closed immediately upon startup, it doesn't seem to affect the reference to the __file__ string (which is not a file object, but a str object)

> 
> 
> a.
> 
> 
> -- 
> I � UTF-8



On Tuesday, July 24, 2012 2:24:31 PM UTC+2, Ervin Hegedüs wrote:
> hello,
> 
> On Tue, Jul 24, 2012 at 04:48:42AM -0700, ivdneut@gmail.com wrote:
> > Hello,
> > 
> > I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
> > 
> > Exception info: Traceback (most recent call last):
> >   File "scheduler.py", line 376, in applyrule
> >     result = execrule(rule_code)
> >   File "scheduler.py", line 521, in execrule
> >     rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
> > NameError: name '__file__' is not defined
> > 
> > This section of the code is executed in this process *all the time*, but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.
> 
> could you send the relevant part of the code?
> 
> I mean: how do you daemonize your process?
>  
> > I am running python from a virtual-env installation from a stock Red Hat EL 6.2 installation:
> > 
> > (virtual-env)[user@host ~]$ python --version
> > Python 2.6.6
> > (virtual-env)[user@host ~]$ cat /etc/redhat-release 
> > Red Hat Enterprise Linux Server release 6.2 (Santiago)
> 
> If you use fork(), it drops all file descriptors, and creates new
> ones - may be then loss the __file__...?
> 
> 
> a.
> 
> 
> -- 
> I � UTF-8

[toc] | [prev] | [next] | [standalone]

#25987

On Tuesday, July 24, 2012 2:24:31 PM UTC+2, Ervin Hegedüs wrote:
> hello,
> 
> On Tue, Jul 24, 2012 at 04:48:42AM -0700, ivdneut@gmail.com wrote:
> > Hello,
> > 
> > I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
> > 
> > Exception info: Traceback (most recent call last):
> >   File "scheduler.py", line 376, in applyrule
> >     result = execrule(rule_code)
> >   File "scheduler.py", line 521, in execrule
> >     rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
> > NameError: name '__file__' is not defined
> > 
> > This section of the code is executed in this process *all the time*, but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.
> 
> could you send the relevant part of the code?
> 
> I mean: how do you daemonize your process?

It's done by a double fork:

## First fork()
pid = os.fork()
if pid != 0: sys.exit(0) # parent exits.

## create new session
os.setsid()
	   
## ignore SIGHUP
signal.signal(signal.SIGHUP, signal.SIG_IGN)

## Second fork()
pid = os.fork()
if pid != 0: sys.exit(0) # First child exits.
	   
## Change working directory to the home directory.
homedir = pwd.getpwuid(os.geteuid())[5]
os.chdir(homedir)

os.umask(0)

for fd in range(0, 1024):
try:
    os.close(fd)
except:
    pass # fd not open, ignore this exception.

The original C version of this code is from W.R. Stevens' daemon_init() routine in "UNIX Network Programming Volume 1, second edition"

>  
> > I am running python from a virtual-env installation from a stock Red Hat EL 6.2 installation:
> > 
> > (virtual-env)[user@host ~]$ python --version
> > Python 2.6.6
> > (virtual-env)[user@host ~]$ cat /etc/redhat-release 
> > Red Hat Enterprise Linux Server release 6.2 (Santiago)
> 
> If you use fork(), it drops all file descriptors, and creates new
> ones - may be then loss the __file__...?

I doubt this would be it, or it would stop working immediately, since daemonization is done upon startup of the process. File descriptors are closed immediately upon startup, it doesn't seem to affect the reference to the __file__ string (which is not a file object, but a str object)

> 
> 
> a.
> 
> 
> -- 
> I � UTF-8



On Tuesday, July 24, 2012 2:24:31 PM UTC+2, Ervin Hegedüs wrote:
> hello,
> 
> On Tue, Jul 24, 2012 at 04:48:42AM -0700, ivdneut@gmail.com wrote:
> > Hello,
> > 
> > I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
> > 
> > Exception info: Traceback (most recent call last):
> >   File "scheduler.py", line 376, in applyrule
> >     result = execrule(rule_code)
> >   File "scheduler.py", line 521, in execrule
> >     rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
> > NameError: name '__file__' is not defined
> > 
> > This section of the code is executed in this process *all the time*, but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.
> 
> could you send the relevant part of the code?
> 
> I mean: how do you daemonize your process?
>  
> > I am running python from a virtual-env installation from a stock Red Hat EL 6.2 installation:
> > 
> > (virtual-env)[user@host ~]$ python --version
> > Python 2.6.6
> > (virtual-env)[user@host ~]$ cat /etc/redhat-release 
> > Red Hat Enterprise Linux Server release 6.2 (Santiago)
> 
> If you use fork(), it drops all file descriptors, and creates new
> ones - may be then loss the __file__...?
> 
> 
> a.
> 
> 
> -- 
> I � UTF-8

[toc] | [prev] | [next] | [standalone]

#25984

> If you use fork(), it drops all file descriptors, and creates new
> ones - may be then loss the __file__...?
>
I don't think this is the case. He wrote that the process runs for weeks 
without problems, and code using __file__ is being executed all the time.

[toc] | [prev] | [next] | [standalone]

#25994

"ivdneut@gmail.com" <ivdneut@gmail.com> writes:

> I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
>
> Exception info: Traceback (most recent call last):
>   File "scheduler.py", line 376, in applyrule
>     result = execrule(rule_code)
>   File "scheduler.py", line 521, in execrule
>     rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
> NameError: name '__file__' is not defined
>
> This section of the code is executed in this process *all the time*, but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.

This is strange indeed.

I have only one vague idea: should something try to terminate the
process, modules would start to lose their variables during shutdown.

[toc] | [prev] | [next] | [standalone]

#26004

Dieter Maurer <dieter@handshake.de> writes:
> I have only one vague idea: should something try to terminate the
> process, modules would start to lose their variables during shutdown.

That happens all the time with multi-threaded programs, because the
shutdown is happening concurrently with other threads doing stuff.  Are
there threads in this particular program?

[toc] | [prev] | [next] | [standalone]

#26008

On Tue, Jul 24, 2012 at 1:32 PM, Paul Rubin <no.email@nospam.invalid> wrote:
> Dieter Maurer <dieter@handshake.de> writes:
>> I have only one vague idea: should something try to terminate the
>> process, modules would start to lose their variables during shutdown.
>
> That happens all the time with multi-threaded programs, because the
> shutdown is happening concurrently with other threads doing stuff.  Are
> there threads in this particular program?

It also comes up in single-threaded programs that use finalizers
(__del__ methods).  At the time an object is finalized, many globals
might already be gone.

[toc] | [prev] | [next] | [standalone]

#26006

On Tue, Jul 24, 2012 at 8:59 AM, Dieter Maurer <dieter@handshake.de> wrote:
> "ivdneut@gmail.com" <ivdneut@gmail.com> writes:
>
>> I have a daemon process that runs for a considerable amount of time (weeks on end) without any problems. At some point I start getting the exception:
>>
>> Exception info: Traceback (most recent call last):
>>   File "scheduler.py", line 376, in applyrule
>>     result = execrule(rule_code)
>>   File "scheduler.py", line 521, in execrule
>>     rulepath = os.path.dirname(__file__)+"/"+'/'.join(rule['modules'])+"/"+rule['rulename']
>> NameError: name '__file__' is not defined
>>
>> This section of the code is executed in this process *all the time*, but suddenly stops working. I have been searching for similar issues online, but only come accross people having problems because they run the script interactively. This is not the case here.
>
> This is strange indeed.
>
> I have only one vague idea: should something try to terminate the
> process, modules would start to lose their variables during shutdown.

That's a good theory.  Or perhaps something in the code itself is
handling the module's globals() and very occasionally does something
that is incorrect and destructive.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web