Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #40447 > unrolled thread

Different behavior with multiprocessing

Back to article view | Back to comp.lang.python

Contents

  Different behavior with multiprocessing Tomas Kotal <tomas.kotal@gmail.com> - 2013-03-04 06:31 -0800
    Re: Different behavior with multiprocessing Chris Angelico <rosuav@gmail.com> - 2013-03-05 01:58 +1100
      Re: Different behavior with multiprocessing Tomas Kotal <tomas.kotal@gmail.com> - 2013-03-04 07:12 -0800
      Re: Different behavior with multiprocessing Tomas Kotal <tomas.kotal@gmail.com> - 2013-03-04 07:12 -0800
    Re: Different behavior with multiprocessing Tomas Kotal <tomas.kotal@gmail.com> - 2013-03-04 07:35 -0800
      Re: Different behavior with multiprocessing Chris Angelico <rosuav@gmail.com> - 2013-03-05 02:44 +1100

#40447 — Different behavior with multiprocessing

Hi all.

Recently I discovered a strange behavior with multiprocessing library and call to function os.system (a different behavior under Linux and Windows to be more specific). I have this simple testing script:

############################################
import sys
import os
from multiprocessing import Process

def do_fork(cmd):
    ret = os.system(cmd)
    # print result of call
    print ret
    os._exit(ret)

if __name__ == "__main__":
    cmds = [ "dir", "xy" ]

    procs = []
    for cmd in cmds:
        proc = Process(target=do_fork, args=(cmd, ))
        proc.start()
        procs.append( proc )

    for proc in procs:
        proc.join()
        print "exitcode: %d" % proc.exitcode

    print "ok" 
############################################

This script just starts 2 processes. Each process executes one command in system shell and exits setting it's exit code same as exit code of a system call. There are 2 commands: "dir" (which works fine on Windows and Linux) and "xy" (which is supposed to fail under both systems). The strange thing is that when I run this script under Windows I get this output:

0
1
exitcode: 0
exitcode: 1
ok

The first 0 and 1 are results of os.system call which are printed from child processes. Rest of lines are printed from main process. This is expected output.

But when I run same script on Linux, what I get is this:

0
32512
exitcode: 0
exitcode: 0
ok

Although the second command fails and returns exit code 32512, the exit code from process in parent process gives me 0. I tried to change the script to use fork() instead of Process but the result was same.

Can anybody explain me what's the problem here?

(I use Python 2.6 on both Windows and Linux machines)

[toc] | [next] | [standalone]

#40449

On Tue, Mar 5, 2013 at 1:31 AM, Tomas Kotal <tomas.kotal@gmail.com> wrote:
> But when I run same script on Linux, what I get is this:
>
> 0
> 32512

Under Unix, the return value from os.system() encodes more than one
piece of information:

http://docs.python.org/2/library/os.html#os.system
http://docs.python.org/2/library/os.html#os.wait

32512 is 127*256, meaning that the shell exited with return code 127
when given an unrecognized command.

ChrisA

[toc] | [prev] | [next] | [standalone]

#40452

> Under Unix, the return value from os.system() encodes more than one
> 
> piece of information:
> 
> 
> 
> http://docs.python.org/2/library/os.html#os.system
> 
> http://docs.python.org/2/library/os.html#os.wait
> 
> 
> 
> 32512 is 127*256, meaning that the shell exited with return code 127
> 
> when given an unrecognized command.
> 
> 
> 
> ChrisA

Well the point is not the difference between return values of os.system under Windows and Linux.

The point is that I set the exist code of child process with os._exit() which works fine on Windows but on Linux the process.exitcode returns always 0.

What is very strange is, that if I set return code to some other value, like:

...
os._exit( len(cmd) )
...

Then I got exit codes 3 and 2 under Linux, which is correct. But when I do something like this:

...
ret = os.system(cmd)
print ret # this print real value, like 32512
os._exit(ret) # exitcode of this child process will be 0 on Linux
...

It doesn't work, which seems very strange to me.

[toc] | [prev] | [next] | [standalone]

#40454

> Under Unix, the return value from os.system() encodes more than one
> 
> piece of information:
> 
> 
> 
> http://docs.python.org/2/library/os.html#os.system
> 
> http://docs.python.org/2/library/os.html#os.wait
> 
> 
> 
> 32512 is 127*256, meaning that the shell exited with return code 127
> 
> when given an unrecognized command.
> 
> 
> 
> ChrisA

Well the point is not the difference between return values of os.system under Windows and Linux.

The point is that I set the exist code of child process with os._exit() which works fine on Windows but on Linux the process.exitcode returns always 0.

What is very strange is, that if I set return code to some other value, like:

...
os._exit( len(cmd) )
...

Then I got exit codes 3 and 2 under Linux, which is correct. But when I do something like this:

...
ret = os.system(cmd)
print ret # this print real value, like 32512
os._exit(ret) # exitcode of this child process will be 0 on Linux
...

It doesn't work, which seems very strange to me.

[toc] | [prev] | [next] | [standalone]

#40456

Dne pondělí, 4. března 2013 15:31:41 UTC+1 Tomas Kotal napsal(a):
> Hi all.
> 
> 
> 
> Recently I discovered a strange behavior with multiprocessing library and call to function os.system (a different behavior under Linux and Windows to be more specific). I have this simple testing script:
> 
> 
> 
> ############################################
> 
> import sys
> 
> import os
> 
> from multiprocessing import Process
> 
> 
> 
> def do_fork(cmd):
> 
>     ret = os.system(cmd)
> 
>     # print result of call
> 
>     print ret
> 
>     os._exit(ret)
> 
> 
> 
> if __name__ == "__main__":
> 
>     cmds = [ "dir", "xy" ]
> 
> 
> 
>     procs = []
> 
>     for cmd in cmds:
> 
>         proc = Process(target=do_fork, args=(cmd, ))
> 
>         proc.start()
> 
>         procs.append( proc )
> 
> 
> 
>     for proc in procs:
> 
>         proc.join()
> 
>         print "exitcode: %d" % proc.exitcode
> 
> 
> 
>     print "ok" 
> 
> ############################################
> 
> 
> 
> This script just starts 2 processes. Each process executes one command in system shell and exits setting it's exit code same as exit code of a system call. There are 2 commands: "dir" (which works fine on Windows and Linux) and "xy" (which is supposed to fail under both systems). The strange thing is that when I run this script under Windows I get this output:
> 
> 
> 
> 0
> 
> 1
> 
> exitcode: 0
> 
> exitcode: 1
> 
> ok
> 
> 
> 
> The first 0 and 1 are results of os.system call which are printed from child processes. Rest of lines are printed from main process. This is expected output.
> 
> 
> 
> But when I run same script on Linux, what I get is this:
> 
> 
> 
> 0
> 
> 32512
> 
> exitcode: 0
> 
> exitcode: 0
> 
> ok
> 
> 
> 
> Although the second command fails and returns exit code 32512, the exit code from process in parent process gives me 0. I tried to change the script to use fork() instead of Process but the result was same.
> 
> 
> 
> Can anybody explain me what's the problem here?
> 
> 
> 
> (I use Python 2.6 on both Windows and Linux machines)

Seems like I found the problem: os._exit probably takes as parametr unsigned char, so it uses as error code whatever value it gets modulo 256:

os._exit(1)  # process.exitcode == 1
os._exit(255)  # process.exitcode == 255
os._exit(256)  # process.exitcode == 0
os._exit(257)  # process.exitcode == 1
os._exit(32512)  # process.exitcode == 0

So on Linux it's necesary to call something like this:
os._exit( os.system(cmd) >> 8 )

Because the first byte of return value on Linux is number of signal which kills the process and the second one is actual exit code.

http://docs.python.org/2/library/os.html#os.wait

[toc] | [prev] | [next] | [standalone]

#40458

On Tue, Mar 5, 2013 at 2:35 AM, Tomas Kotal <tomas.kotal@gmail.com> wrote:
> Seems like I found the problem: os._exit probably takes as parametr unsigned char, so it uses as error code whatever value it gets modulo 256:
>
> os._exit(1)  # process.exitcode == 1
> os._exit(255)  # process.exitcode == 255
> os._exit(256)  # process.exitcode == 0
> os._exit(257)  # process.exitcode == 1
> os._exit(32512)  # process.exitcode == 0
>
> So on Linux it's necesary to call something like this:
> os._exit( os.system(cmd) >> 8 )
>
> Because the first byte of return value on Linux is number of signal which kills the process and the second one is actual exit code.

Yep. I had a reply part-written but you beat me to it! That is indeed
what you need if you want to chain return values.

However, why are you using os._exit? Check out the note here:

http://docs.python.org/2/library/os.html#os._exit

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web