Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #78448 > unrolled thread

Obscuring Python source from end users

Back to article view | Back to comp.lang.python

Contents

  Obscuring Python source from end users norman.ives@gmail.com - 2014-09-29 00:36 -0700
    Re: Obscuring Python source from end users Chris Angelico <rosuav@gmail.com> - 2014-09-29 17:52 +1000
      Re: Obscuring Python source from end users norman.ives@gmail.com - 2014-09-29 01:41 -0700
        Re: Obscuring Python source from end users Chris Angelico <rosuav@gmail.com> - 2014-09-29 18:55 +1000
        Re: Obscuring Python source from end users Chris Angelico <rosuav@gmail.com> - 2014-09-29 20:14 +1000
    Re: Obscuring Python source from end users alister <alister.nospam.ware@ntlworld.com> - 2014-09-29 08:42 +0000
    Re: Obscuring Python source from end users Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-09-29 21:47 +1000
      Re: Obscuring Python source from end users Sturla Molden <sturla.molden@gmail.com> - 2014-09-29 14:51 +0000
      Re: Obscuring Python source from end users Dan Stromberg <drsalists@gmail.com> - 2014-10-01 15:06 -0700
        Re: Obscuring Python source from end users Marko Rauhamaa <marko@pacujo.net> - 2014-10-02 08:17 +0300
          Re: Obscuring Python source from end users Terry Reedy <tjreedy@udel.edu> - 2014-10-02 03:19 -0400
          Re: Obscuring Python source from end users Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-02 17:22 +1000
    Re: Obscuring Python source from end users Sturla Molden <sturla.molden@gmail.com> - 2014-09-29 14:53 +0000
      Re: Obscuring Python source from end users Johannes Bauer <dfnsonfsduifb@gmx.de> - 2014-10-02 09:50 +0200
        Re: Obscuring Python source from end users Chris Angelico <rosuav@gmail.com> - 2014-10-02 18:32 +1000

#78448 — Obscuring Python source from end users

Hello list

Python 3.4 applies.

I have a project that involves distributing Python code to users in an organisation. Users do not interact directly with the Python code; they only know this project as an Excel add-in.

Now, internal audit takes exception in some cases if users are able to see the source code.

So I'm wondering if anyone has clever suggestions in this regard...

My current plan is to modify the bdist_wheel setuptools extension so that it can produce distributions with only the .pyc files, laid out so that they will be importable in the normal way. This will be sufficient to satisfy internal audit, and will not negatively impact our users.

However there are obvious downsides, including the overhead of maintaining separate wheels for different architectures and (in the future) Python versions. Not to mention that this plan seems to go against the grain of how Python wants to use byte code files...

Is there a better solution?

[toc] | [next] | [standalone]

#78449

On Mon, Sep 29, 2014 at 5:36 PM,  <norman.ives@gmail.com> wrote:
> I have a project that involves distributing Python code to users in an organisation. Users do not interact directly with the Python code; they only know this project as an Excel add-in.
>
> Now, internal audit takes exception in some cases if users are able to see the source code.

The solution is to fix your internal audit. There's fundamentally no
way to hide the source code, and it's going to add complexity.
Demonstrate that it could take you a large number of man-hours and
achieve little, and just declare that the source code IS what you
want.

Alternatively, you could do a source-level reformat that crunches
everything down as much as possible, while still producing
syntactically-identical source code. Remove all the comments, shorten
local names to one letter, etc. That would result in something that,
while still perfectly executable, won't be nearly as helpful. It
wouldn't be nice readable source code, at least. Would your audit be
satisfied with that?

ChrisA

[toc] | [prev] | [next] | [standalone]

#78451

Thanks for the reply!

I'm not concerned about hiding the source code in a fundamental way. The condition that needs to be satisfied is that independent validators (in the organisation) must not "have access" to the source code.

Crunching the source is an interesting idea that could achieve that end, but it seems fraught with problems like maintaining consistency between renaming objects in a module and renaming where imports happen.

[toc] | [prev] | [next] | [standalone]

#78453

On Mon, Sep 29, 2014 at 6:41 PM,  <norman.ives@gmail.com> wrote:
> Crunching the source is an interesting idea that could achieve that end, but it seems fraught with problems like maintaining consistency between renaming objects in a module and renaming where imports happen.
>

Here's a technique that you could use. Pick two prefixes, one for long
names and one for short names, and maintain a translation table. So,
for instance, you could have two source files like this:

# file1.py
# Do something with the length and the count
def xx_some_func(length, cnt):
    # TODO: Implement me.
    pass

# file2.py
import file1

def xx_foo(lst, len, count):
    # Do something for every element in the list
    # where the something is blah blah blah
    for element in lst:
        file1.xx_some_func(len, count)


And then your crunched files might be:

# file1.py:
def x_a(a,b):
 pass

# file2.py
import file1
def x_b(a,b,c):
 for d in a:
  file2.x_a(b,c)


The translation is 100% mechanical: any place you find "xx_some_func",
you replace it with "x_a". If you find a string that begins "xx_" and
isn't in your translation table, you construct a new name (don't
forget that you can use x_A as well as x_a) and add it to the table.

It ought to be possible to do an AST reconstitution for at least part
of this. I can hunt down some of my PEP 463 test code to help out with
that. It should be possible to figure out what names are local, and
then just use those.

If this is of interest, I'll see what I can knock together.

ChrisA

[toc] | [prev] | [next] | [standalone]

#78456

On Mon, Sep 29, 2014 at 6:55 PM, Chris Angelico <rosuav@gmail.com> wrote:
> It ought to be possible to do an AST reconstitution for at least part
> of this. I can hunt down some of my PEP 463 test code to help out with
> that. It should be possible to figure out what names are local, and
> then just use those.
>
> If this is of interest, I'll see what I can knock together.

Actually, why reconstitute? The AST can be compiled.

Here's a Python cruncher/obscurer:

import ast, sys, os

# Note: If the file can't be parsed, this will raise SyntaxError
# (from the ast.parse() line).
def crunch(fn, outfn):
    with open(fn, "rb") as f:
        data = f.read()
    tree = ast.parse(data)
    os.makedirs(os.path.split(outfn)[0], exist_ok=True)
    with open(outfn, "w") as f:
        print("from ast import *", file=f)
        print("_code = "+ast.dump(tree), file=f)
        print("fix_missing_locations(_code)", file=f)
        print("[globals().__delitem__(_) for _ in dir() if _[0]!='_']", file=f)
        print("exec(compile(_code,__name__,'exec'))", file=f)

if __name__ == "__main__":
    for fn in sys.argv[1:]:
        outfn = "crunched/" + fn
        crunch(fn, outfn)
        print("Crunched", fn, "to", outfn)


The resulting files should work just the same as the original ones,
but will be virtually unreadable. This is still only obscurity,
though; the only information actually removed is formatting and
comments, not even names. But you could easily add an ast.walk() in
there, just after the ast.parse(), and do whatever transformations you
want.

Note that unlike with JavaScript crunching, this won't shrink the
files - in fact, it's usually going to expand them. They'll probably
also take longer to load. There is no benefit other than the
obscurity.

Run this over some of your files, then see if your auditors are happy.

ChrisA

[toc] | [prev] | [next] | [standalone]

#78452

On Mon, 29 Sep 2014 00:36:47 -0700, norman.ives wrote:

> Hello list
> 
> Python 3.4 applies.
> 
> I have a project that involves distributing Python code to users in an
> organisation. Users do not interact directly with the Python code; they
> only know this project as an Excel add-in.
> 
> Now, internal audit takes exception in some cases if users are able to
> see the source code.
> 
> So I'm wondering if anyone has clever suggestions in this regard...
> 
> My current plan is to modify the bdist_wheel setuptools extension so
> that it can produce distributions with only the .pyc files, laid out so
> that they will be importable in the normal way. This will be sufficient
> to satisfy internal audit, and will not negatively impact our users.
> 
> However there are obvious downsides, including the overhead of
> maintaining separate wheels for different architectures and (in the
> future) Python versions. Not to mention that this plan seems to go
> against the grain of how Python wants to use byte code files...
> 
> Is there a better solution?

For an internal app. I would suggest it is only necessary to protect user-
names & passwords to back end services (databases etc.).

storing these in a separate file in an encrypted format should be enough 
to discourage internal users from digging deeper.

This will not stop a determined hacker but neither will any other forms 
of obfuscation.



-- 
new, adj.:
	Different color from previous model.

[toc] | [prev] | [next] | [standalone]

#78458

norman.ives@gmail.com wrote:

> Hello list
> 
> Python 3.4 applies.
> 
> I have a project that involves distributing Python code to users in an
> organisation. Users do not interact directly with the Python code; they
> only know this project as an Excel add-in.
> 
> Now, internal audit takes exception in some cases if users are able to see
> the source code.

You have my sympathy.


> So I'm wondering if anyone has clever suggestions in this regard...

Yes. Distribute the pyc files only. That is the canonical Python answer to
the problem of not distributing source code. You may need a tiny "driver"
script (or perhaps not), if you do it will be something as minor as:

import module_where_all_the_work_is_really_done as module
module.main()

depending on how the Excel add-in system works.

> My current plan is to modify the bdist_wheel setuptools extension so that
> it can produce distributions with only the .pyc files, laid out so that
> they will be importable in the normal way. This will be sufficient to
> satisfy internal audit, and will not negatively impact our users.

Sounds good to me. Do you know about the compileall.py script/module in the
standard library?


> However there are obvious downsides, including the overhead of maintaining
> separate wheels for different architectures and (in the future) Python
> versions. Not to mention that this plan seems to go against the grain of
> how Python wants to use byte code files...

Well, yes, it does go against the grain, but byte-code only distributions
are officially supported. Occasionally people have made requests to
simplify the import system by dropping support for .pyc only imports, but
Python's creator Guido van Rossum has made it clear that as alien as such a
thing is to the open source community, Python is going to allow it.

Another possibility is to distribute your modules inside a zip file. See
here:

https://mail.python.org/pipermail/python-list/2014-July/675506.html

Such zip files are not just runnable, but also importable. Depending on your
Excel requirements, you might need a tiny driver script as above.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#78467

Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> Another possibility is to distribute your modules inside a zip file. See
> here:
> 
> https://mail.python.org/pipermail/python-list/2014-July/675506.html
> 
> Such zip files are not just runnable, but also importable. Depending on your
> Excel requirements, you might need a tiny driver script as above.

It should also be possible to temper with the zipimport module to allow a
'scrambled' zip file. That makes it harder for users to access the code by
unpacking the zip file.

py2exe will collect pyc files into a single zip file and import that. It
does not need a driver script because it embeds the Python interpreter in
an executable. 

Yet another option is to compile the Python code with Cython and distribute
everything as compiled pyd files.

Sturla

[toc] | [prev] | [next] | [standalone]

#78566

On Mon, Sep 29, 2014 at 4:47 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> norman.ives@gmail.com wrote:
>
>> Hello list
>>
>> Python 3.4 applies.
>>
>> I have a project that involves distributing Python code to users in an
>> organisation. Users do not interact directly with the Python code; they
>> only know this project as an Excel add-in.
>>
>> Now, internal audit takes exception in some cases if users are able to see
>> the source code.
>
> You have my sympathy.

Actually, I don't think this is that bad.  Companies have rules; one
simply has to learn to live with this or suffer greatly.  Some rules
are worth fighting, but this probably isn't one of them.

>> So I'm wondering if anyone has clever suggestions in this regard...
>
> Yes. Distribute the pyc files only.

Yes, this is the way it's usually done.  Supposedly Microsoft did this
for a while, after acquiring some Python software.  In the Python 3.4
case, the pyc's should be in the __pycache__ subdirectory.

Another option would be to convert to Cython and compile (not as bad
as it sounds).  But __pycache__/*.pyc would probably be easier.

[toc] | [prev] | [next] | [standalone]

#78578

Dan Stromberg <drsalists@gmail.com>:

> On Mon, Sep 29, 2014 at 4:47 AM, Steven D'Aprano
>> Yes. Distribute the pyc files only.
>
> Yes, this is the way it's usually done.

Has the .pyc file format stabilized? A decade ago, my employer shipped
an application as .pyc files but had to ship the matching CPython binary
with it.


Marko

[toc] | [prev] | [next] | [standalone]

#78582

On 10/2/2014 1:17 AM, Marko Rauhamaa wrote:
> Dan Stromberg <drsalists@gmail.com>:
>
>> On Mon, Sep 29, 2014 at 4:47 AM, Steven D'Aprano
>>> Yes. Distribute the pyc files only.
>>
>> Yes, this is the way it's usually done.
>
> Has the .pyc file format stabilized?

No. The cache files are binary specific and are so labelled. x.py is 
compiled, for instance, to x.cpython-34.pyc

> A decade ago, my employer shippe
> an application as .pyc files but had to ship the matching CPython binary
> with it.


-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#78583

Marko Rauhamaa wrote:

> Dan Stromberg <drsalists@gmail.com>:
> 
>> On Mon, Sep 29, 2014 at 4:47 AM, Steven D'Aprano
>>> Yes. Distribute the pyc files only.
>>
>> Yes, this is the way it's usually done.
> 
> Has the .pyc file format stabilized? A decade ago, my employer shipped
> an application as .pyc files but had to ship the matching CPython binary
> with it.

There is no promise that Python byte code or the format of .pyc files will
be stable across minor releases. However, PEP 6 states that bugfix releases
should have stable .pyc files. 

http://legacy.python.org/dev/peps/pep-0006/

So, you ought to expect that .pyc files will change when any of the
following differs:

- the hardware (e.g. 32- versus 64-bit, or ARM versus x86 processor);

- the Python implementation (e.g. Stackless versus CPython);

- the minor or major version number (e.g. 2.6 to 2.7, or 2.x to 3.x);

but you can expect the .pyc files to remain compatible for:

- the operating system (e.g. Windows, Mac, Linux);

- bug-release versions (e.g. 2.7.1 to 2.7.2).

Of course, you can distribute the Python executable, but it may be easier to
just distribute a separate .pyc file for each minor release you wish to
support.

Here are some comments from one of the senior core developers:

http://www.curiousefficiency.org/posts/2011/04/benefits-and-limitations-of-pyc-only.html



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#78468

Chris Angelico <rosuav@gmail.com> wrote:

>> I have a project that involves distributing Python code to users in an
>> organisation. Users do not interact directly with the Python code; they
>> only know this project as an Excel add-in.
>> 
>> Now, internal audit takes exception in some cases if users are able to
>> see the source code.
> 
> The solution is to fix your internal audit.

+1

[toc] | [prev] | [next] | [standalone]

#78584

On 29.09.2014 16:53, Sturla Molden wrote:
> Chris Angelico <rosuav@gmail.com> wrote:
> 
>>> I have a project that involves distributing Python code to users in an
>>> organisation. Users do not interact directly with the Python code; they
>>> only know this project as an Excel add-in.
>>>
>>> Now, internal audit takes exception in some cases if users are able to
>>> see the source code.
>>
>> The solution is to fix your internal audit.
> 
> +1

You two have obviously never worked at a large corporation or you would
both realize how tremendously naive and unhelpful this "solution" is.

Cheers,
Johannes

-- 
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht öffentlich!
Ah, der neueste und bis heute genialste Streich unsere großen
Kosmologen: Die Geheim-Vorhersage.
 - Karl Kaos über Rüdiger Thomas in dsa <hidbv3$om2$1@speranza.aioe.org>

[toc] | [prev] | [next] | [standalone]

#78588

On Thu, Oct 2, 2014 at 5:50 PM, Johannes Bauer <dfnsonfsduifb@gmx.de> wrote:
> On 29.09.2014 16:53, Sturla Molden wrote:
>> Chris Angelico <rosuav@gmail.com> wrote:
>>
>>>> I have a project that involves distributing Python code to users in an
>>>> organisation. Users do not interact directly with the Python code; they
>>>> only know this project as an Excel add-in.
>>>>
>>>> Now, internal audit takes exception in some cases if users are able to
>>>> see the source code.
>>>
>>> The solution is to fix your internal audit.
>>
>> +1
>
> You two have obviously never worked at a large corporation or you would
> both realize how tremendously naive and unhelpful this "solution" is.

Unhelpful? Perhaps, perhaps not. I would, however, distinguish between
the two parts of my original response as follows:

1) The *solution* is to fix the audit process.

2) But here's one of several *work-arounds*.

It may well be impossible to truly solve the problem, and I'm aware of
that. But sometimes you need to point out that a pyc-only distribution
is still distributing something very VERY close to the source code,
and therefore it's nigh useless to talk of not distributing Python
source.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web