Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #10336 > unrolled thread

Only Bytecode, No .py Files

Back to article view | Back to comp.lang.python

Contents

  Only Bytecode, No .py Files Eldon Ziegler <eldonz@atlanticdb.com> - 2011-07-26 11:19 -0400
    Re: Only Bytecode, No .py Files Billy Mays <81282ed9a88799d21e77957df2d84bd6514d9af6@myhashismyemail.com> - 2011-07-26 11:27 -0400
    Re: Only Bytecode, No .py Files Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-07-27 09:10 +0200
      Re: Only Bytecode, No .py Files John Roth <johnroth1@gmail.com> - 2011-07-27 05:18 -0700
        Re: Only Bytecode, No .py Files Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-07-27 16:56 +0200
          Re: Only Bytecode, No .py Files John Roth <johnroth1@gmail.com> - 2011-07-29 05:51 -0700
            Re: Only Bytecode, No .py Files Ethan Furman <ethan@stoneleaf.us> - 2011-07-29 07:09 -0700
            Re: Only Bytecode, No .py Files Jerry Hill <malaclypse2@gmail.com> - 2011-07-29 10:48 -0400
            Re: Only Bytecode, No .py Files Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-07-29 16:47 +0200
    Re: Only Bytecode, No .py Files Alan Meyer <ameyer2@yahoo.com> - 2011-07-29 11:06 -0400

#10336 — Only Bytecode, No .py Files

Is there a way to have the Python processor look only for bytecode
files, not .py files? We are seeing huge numbers of Linux audit messages
on production system on which only bytecode files are stored. The audit
subsystem is recording each open failure.

Thanks,
Eldon Ziegler

[toc] | [next] | [standalone]

#10338

On 07/26/2011 11:19 AM, Eldon Ziegler wrote:
> Is there a way to have the Python processor look only for bytecode
> files, not .py files? We are seeing huge numbers of Linux audit messages
> on production system on which only bytecode files are stored. The audit
> subsystem is recording each open failure.
>
> Thanks,
> Eldon Ziegler
>
>

How are you opening your files?

--
Bill

[toc] | [prev] | [next] | [standalone]

#10375

Am 26.07.2011 17:19 schrieb Eldon Ziegler:
> Is there a way to have the Python processor look only for bytecode
> files, not .py files? We are seeing huge numbers of Linux audit messages
> on production system on which only bytecode files are stored. The audit
> subsystem is recording each open failure.

Is that really a problem? AFAIK there are many failing open() calls on 
the start of every program.

E.g.

open("/lib/bash/4.1/tls/i686/sse2/libncurses.so.5", O_RDONLY) = -1 
ENOENT (No such file or directory)
stat64("/lib/bash/4.1/tls/i686/sse2", 0xbfd3a350) = -1 ENOENT (No such 
file or directory)
open("/lib/bash/4.1/tls/i686/libncurses.so.5", O_RDONLY) = -1 ENOENT (No 
such file or directory)
stat64("/lib/bash/4.1/tls/i686", 0xbfd3a350) = -1 ENOENT (No such file 
or directory)
open("/lib/bash/4.1/tls/sse2/libncurses.so.5", O_RDONLY) = -1 ENOENT (No 
such file or directory)
stat64("/lib/bash/4.1/tls/sse2", 0xbfd3a350) = -1 ENOENT (No such file 
or directory)
open("/lib/bash/4.1/tls/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such 
file or directory)
stat64("/lib/bash/4.1/tls", 0xbfd3a350) = -1 ENOENT (No such file or 
directory)
open("/lib/bash/4.1/i686/sse2/libncurses.so.5", O_RDONLY) = -1 ENOENT 
(No such file or directory)
stat64("/lib/bash/4.1/i686/sse2", 0xbfd3a350) = -1 ENOENT (No such file 
or directory)
open("/lib/bash/4.1/i686/libncurses.so.5", O_RDONLY) = -1 ENOENT (No 
such file or directory)
stat64("/lib/bash/4.1/i686", 0xbfd3a350) = -1 ENOENT (No such file or 
directory)
open("/lib/bash/4.1/sse2/libncurses.so.5", O_RDONLY) = -1 ENOENT (No 
such file or directory)
stat64("/lib/bash/4.1/sse2", 0xbfd3a350) = -1 ENOENT (No such file or 
directory)
open("/lib/bash/4.1/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such 
file or directory)
stat64("/lib/bash/4.1", 0xbfd3a350)     = -1 ENOENT (No such file or 
directory)
open("/lib/libncurses.so.5", O_RDONLY)  = 3

is a part of what happens if I start the MySQL client,

Even starting the bash results in

open("/lib/bash/4.1/tls/i686/sse2/libreadline.so.6", O_RDONLY) = -1 
ENOENT (No such file or directory)
stat64("/lib/bash/4.1/tls/i686/sse2", 0xbfe0c4d0) = -1 ENOENT (No such 
file or directory)
open("/lib/bash/4.1/tls/i686/libreadline.so.6", O_RDONLY) = -1 ENOENT 
(No such file or directory)
stat64("/lib/bash/4.1/tls/i686", 0xbfe0c4d0) = -1 ENOENT (No such file 
or directory)
open("/lib/bash/4.1/tls/sse2/libreadline.so.6", O_RDONLY) = -1 ENOENT 
(No such file or directory)
stat64("/lib/bash/4.1/tls/sse2", 0xbfe0c4d0) = -1 ENOENT (No such file 
or directory)
open("/lib/bash/4.1/tls/libreadline.so.6", O_RDONLY) = -1 ENOENT (No 
such file or directory)
stat64("/lib/bash/4.1/tls", 0xbfe0c4d0) = -1 ENOENT (No such file or 
directory)
open("/lib/bash/4.1/i686/sse2/libreadline.so.6", O_RDONLY) = -1 ENOENT 
(No such file or directory)
stat64("/lib/bash/4.1/i686/sse2", 0xbfe0c4d0) = -1 ENOENT (No such file 
or directory)
open("/lib/bash/4.1/i686/libreadline.so.6", O_RDONLY) = -1 ENOENT (No 
such file or directory)
stat64("/lib/bash/4.1/i686", 0xbfe0c4d0) = -1 ENOENT (No such file or 
directory)
open("/lib/bash/4.1/sse2/libreadline.so.6", O_RDONLY) = -1 ENOENT (No 
such file or directory)
stat64("/lib/bash/4.1/sse2", 0xbfe0c4d0) = -1 ENOENT (No such file or 
directory)
open("/lib/bash/4.1/libreadline.so.6", O_RDONLY) = -1 ENOENT (No such 
file or directory)
stat64("/lib/bash/4.1", 0xbfe0c4d0)     = -1 ENOENT (No such file or 
directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3

So can it really be such a huge problem?


Thomas

[toc] | [prev] | [next] | [standalone]

#10384

On Jul 27, 1:10 am, Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-
a470-7603bd3aa...@spamschutz.glglgl.de> wrote:
> Am 26.07.2011 17:19 schrieb Eldon Ziegler:
>
> > Is there a way to have the Python processor look only for bytecode
> > files, not .py files? We are seeing huge numbers of Linux audit messages
> > on production system on which only bytecode files are stored. The audit
> > subsystem is recording each open failure.
>
> Is that really a problem? AFAIK there are many failing open() calls on
> the start of every program.
>
> E.g.
>
> open("/lib/bash/4.1/tls/i686/sse2/libncurses.so.5", O_RDONLY) = -1
> ENOENT (No such file or directory)
> stat64("/lib/bash/4.1/tls/i686/sse2", 0xbfd3a350) = -1 ENOENT (No such
> file or directory)
> open("/lib/bash/4.1/tls/i686/libncurses.so.5", O_RDONLY) = -1 ENOENT (No
> such file or directory)
> stat64("/lib/bash/4.1/tls/i686", 0xbfd3a350) = -1 ENOENT (No such file
> or directory)
> open("/lib/bash/4.1/tls/sse2/libncurses.so.5", O_RDONLY) = -1 ENOENT (No
> such file or directory)
> stat64("/lib/bash/4.1/tls/sse2", 0xbfd3a350) = -1 ENOENT (No such file
> or directory)
> open("/lib/bash/4.1/tls/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such
> file or directory)
> stat64("/lib/bash/4.1/tls", 0xbfd3a350) = -1 ENOENT (No such file or
> directory)
> open("/lib/bash/4.1/i686/sse2/libncurses.so.5", O_RDONLY) = -1 ENOENT
> (No such file or directory)
> stat64("/lib/bash/4.1/i686/sse2", 0xbfd3a350) = -1 ENOENT (No such file
> or directory)
> open("/lib/bash/4.1/i686/libncurses.so.5", O_RDONLY) = -1 ENOENT (No
> such file or directory)
> stat64("/lib/bash/4.1/i686", 0xbfd3a350) = -1 ENOENT (No such file or
> directory)
> open("/lib/bash/4.1/sse2/libncurses.so.5", O_RDONLY) = -1 ENOENT (No
> such file or directory)
> stat64("/lib/bash/4.1/sse2", 0xbfd3a350) = -1 ENOENT (No such file or
> directory)
> open("/lib/bash/4.1/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such
> file or directory)
> stat64("/lib/bash/4.1", 0xbfd3a350)     = -1 ENOENT (No such file or
> directory)
> open("/lib/libncurses.so.5", O_RDONLY)  = 3
>
> is a part of what happens if I start the MySQL client,
>
> Even starting the bash results in
>
> open("/lib/bash/4.1/tls/i686/sse2/libreadline.so.6", O_RDONLY) = -1
> ENOENT (No such file or directory)
> stat64("/lib/bash/4.1/tls/i686/sse2", 0xbfe0c4d0) = -1 ENOENT (No such
> file or directory)
> open("/lib/bash/4.1/tls/i686/libreadline.so.6", O_RDONLY) = -1 ENOENT
> (No such file or directory)
> stat64("/lib/bash/4.1/tls/i686", 0xbfe0c4d0) = -1 ENOENT (No such file
> or directory)
> open("/lib/bash/4.1/tls/sse2/libreadline.so.6", O_RDONLY) = -1 ENOENT
> (No such file or directory)
> stat64("/lib/bash/4.1/tls/sse2", 0xbfe0c4d0) = -1 ENOENT (No such file
> or directory)
> open("/lib/bash/4.1/tls/libreadline.so.6", O_RDONLY) = -1 ENOENT (No
> such file or directory)
> stat64("/lib/bash/4.1/tls", 0xbfe0c4d0) = -1 ENOENT (No such file or
> directory)
> open("/lib/bash/4.1/i686/sse2/libreadline.so.6", O_RDONLY) = -1 ENOENT
> (No such file or directory)
> stat64("/lib/bash/4.1/i686/sse2", 0xbfe0c4d0) = -1 ENOENT (No such file
> or directory)
> open("/lib/bash/4.1/i686/libreadline.so.6", O_RDONLY) = -1 ENOENT (No
> such file or directory)
> stat64("/lib/bash/4.1/i686", 0xbfe0c4d0) = -1 ENOENT (No such file or
> directory)
> open("/lib/bash/4.1/sse2/libreadline.so.6", O_RDONLY) = -1 ENOENT (No
> such file or directory)
> stat64("/lib/bash/4.1/sse2", 0xbfe0c4d0) = -1 ENOENT (No such file or
> directory)
> open("/lib/bash/4.1/libreadline.so.6", O_RDONLY) = -1 ENOENT (No such
> file or directory)
> stat64("/lib/bash/4.1", 0xbfe0c4d0)     = -1 ENOENT (No such file or
> directory)
> open("/etc/ld.so.cache", O_RDONLY)      = 3
>
> So can it really be such a huge problem?
>
> Thomas

Two comments. First, your trace isn't showing attempts to open .py
files, it's showing attempts to open the Curses library in the bash
directory. Maybe you also have a problem with the .py files, but it
isn't documented in this trace. It's also not showing the program
that's causing the failing open.

Second, the audit program is an idiot. There are lots of programs
which use the "easier to ask forgiveness" pattern and test for the
existence of optional files by trying to open them. This may be what
Bash is doing.

John Roth

[toc] | [prev] | [next] | [standalone]

#10388

Am 27.07.2011 14:18 schrieb John Roth:

> Two comments. First, your trace isn't showing attempts to open .py
> files, it's showing attempts to open the Curses library in the bash
> directory.

Of course. My goal was to show that the OP's "problem" is not a python 
one, but occurs all over several programs.


 > Maybe you also have a problem with the .py files,

Why should I?


 > It's also not showing the program that's causing the failing open.

It is irrelevant, IMO.

It just shows that it seems to be common, even for the bin loader, to 
search for libraries to be linked to by trying to open them from several 
places. So every program call must be full of "failures" shown by the 
"audit program".


> Second, the audit program is an idiot. There are lots of programs
> which use the "easier to ask forgiveness" pattern and test for the
> existence of optional files by trying to open them. This may be what
> Bash is doing.

ACK. That is exactly what I wanted to show. (With the difference that 
this is probably nt the bash, but the linux loader trying to link a .so, 
but for the problem, it doesn't make any difference.)


Thomas

[toc] | [prev] | [next] | [standalone]

#10513

On Jul 27, 8:56 am, Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-
a470-7603bd3aa...@spamschutz.glglgl.de> wrote:
> Am 27.07.2011 14:18 schrieb John Roth:
>
> > Two comments. First, your trace isn't showing attempts to open .py
> > files, it's showing attempts to open the Curses library in the bash
> > directory.
>
> Of course. My goal was to show that the OP's "problem" is not a python
> one, but occurs all over several programs.
>
>  > Maybe you also have a problem with the .py files,
>
> Why should I?
>
>  > It's also not showing the program that's causing the failing open.
>
> It is irrelevant, IMO.
>
> It just shows that it seems to be common, even for the bin loader, to
> search for libraries to be linked to by trying to open them from several
> places. So every program call must be full of "failures" shown by the
> "audit program".
>
> > Second, the audit program is an idiot. There are lots of programs
> > which use the "easier to ask forgiveness" pattern and test for the
> > existence of optional files by trying to open them. This may be what
> > Bash is doing.
>
> ACK. That is exactly what I wanted to show. (With the difference that
> this is probably nt the bash, but the linux loader trying to link a .so,
> but for the problem, it doesn't make any difference.)
>
> Thomas

Sorry. I thought what you posted was from the OP. I guess I don't
really expect someone to post a completely irrelevant trace in a
thread started by someone who has a problem.

John Roth

[toc] | [prev] | [next] | [standalone]

#10515

John Roth wrote:
>> ACK. That is exactly what I wanted to show. (With the difference that
>> this is probably nt the bash, but the linux loader trying to link a .so,
>> but for the problem, it doesn't make any difference.)
> 
> Sorry. I thought what you posted was from the OP. I guess I don't
> really expect someone to post a completely irrelevant trace in a
> thread started by someone who has a problem.

The trace was showing the same results in an effort to demonstrate that 
the "problem" was just normal operations.  This is very relevant.

~Ethan~

[toc] | [prev] | [next] | [standalone]

#10518

On Fri, Jul 29, 2011 at 8:51 AM, John Roth <johnroth1@gmail.com> wrote:
> Sorry. I thought what you posted was from the OP. I guess I don't
> really expect someone to post a completely irrelevant trace in a
> thread started by someone who has a problem.

I'm not sure why you would think that that post was by the original
poster, or that it was irrelevant.  It seems to me that it
demonstrated exactly the behavior Eldon was complaining about as a
normal part of the operation of lots of normal unix programs.  Thomas
even said that in the very first line of his post (after the quoted
bit).

-- 
Jerry

[toc] | [prev] | [next] | [standalone]

#10519

Am 29.07.2011 14:51 schrieb John Roth:

> Sorry. I thought what you posted was from the OP. I guess I don't
> really expect someone to post a completely irrelevant trace in a
> thread started by someone who has a problem.

In what way do you find the trace irrelevant? It clearly shows that it 
is not a good idea for an "audit tool" to report all failing open() 
calls and that the OP's "problem" is none, at least not one related to 
Python.

No more, no less.


Thomas

[toc] | [prev] | [next] | [standalone]

#10520

On 07/26/2011 11:19 AM, Eldon Ziegler wrote:
> Is there a way to have the Python processor look only for bytecode
> files, not .py files? We are seeing huge numbers of Linux audit messages
> on production system on which only bytecode files are stored. The audit
> subsystem is recording each open failure.
>
> Thanks,
> Eldon Ziegler

Here are two suggestions:

1. Put a dummy python file in the directory that does very little, 
perhaps something like:

    print("%s: You should not be seeing this" % sys.argv[0])

Give it the name of one of the .pyc, e.g., if foo.pyc, then foo.py.

Give the bytecode file a later date, e.g., "touch foo.pyc" (assuming you 
can do that without messing up other aspects of your system.)

If that works, then copy the dummy file to each of the other required 
.py names in the directory.  You'll then have to touch *.pyc to ensure 
that the pyc's have later dates.  After that, you won't have to do 
anything when you replace a .pyc file.  When you add a new .pyc you'll 
need to do the copy and touch again.

2. Use a log truncator to truncate the audit log.

I don't remember the name of one at the moment, but I believe there is 
at least one open source program that will monitor named files and 
truncate them from the beginning to a fixed maximum size.

I don't like this method as much as the first because it might result in 
something important being truncated.

     Alan

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web