Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #61083 > unrolled thread

Packaging a proprietary Python library for multiple OSs

Back to article view | Back to comp.lang.python

Contents

  Packaging a proprietary Python library for multiple OSs Michael Herrmann <michael.herrmann@heliumhq.com> - 2013-12-05 02:14 -0800
    Re: Packaging a proprietary Python library for multiple OSs rusi <rustompmody@gmail.com> - 2013-12-05 02:56 -0800
      Re: Packaging a proprietary Python library for multiple OSs Travis Griggs <travisgriggs@gmail.com> - 2013-12-05 07:32 -0800
      Re: Packaging a proprietary Python library for multiple OSs Michael Herrmann <michael.herrmann@heliumhq.com> - 2013-12-05 07:41 -0800
      Re: Packaging a proprietary Python library for multiple OSs Chris Angelico <rosuav@gmail.com> - 2013-12-06 03:20 +1100
        Re: Packaging a proprietary Python library for multiple OSs Roy Smith <roy@panix.com> - 2013-12-05 09:12 -0800
          Re: Packaging a proprietary Python library for multiple OSs Chris Angelico <rosuav@gmail.com> - 2013-12-06 04:46 +1100
          Re: Packaging a proprietary Python library for multiple OSs Zero Piraeus <z@etiol.net> - 2013-12-05 14:50 -0300
          Re: Packaging a proprietary Python library for multiple OSs Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-12-05 18:18 +0000
          Re: Packaging a proprietary Python library for multiple OSs Robert Kern <robert.kern@gmail.com> - 2013-12-05 18:28 +0000
      Re: Packaging a proprietary Python library for multiple OSs Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-12-05 17:22 +0000
      Official discussion forum for a project (was: Packaging a proprietary Python library for multiple OSs) Ben Finney <ben+python@benfinney.id.au> - 2013-12-06 08:56 +1100
    Re: Packaging a proprietary Python library for multiple OSs Roy Smith <roy@panix.com> - 2013-12-05 09:09 -0500
      Re: Packaging a proprietary Python library for multiple OSs Michael Herrmann <michael.herrmann@heliumhq.com> - 2013-12-05 07:49 -0800
        Re: Packaging a proprietary Python library for multiple OSs random832@fastmail.us - 2013-12-05 10:52 -0500
          Re: Packaging a proprietary Python library for multiple OSs mherrmann.at@gmail.com - 2013-12-05 08:24 -0800
    Re: Packaging a proprietary Python library for multiple OSs Kevin Walzer <kw@codebykevin.com> - 2013-12-05 10:26 -0500
      Re: Packaging a proprietary Python library for multiple OSs Michael Herrmann <michael.herrmann@heliumhq.com> - 2013-12-05 07:50 -0800
        Re: Packaging a proprietary Python library for multiple OSs Kevin Walzer <kw@codebykevin.com> - 2013-12-05 11:03 -0500
        Re: Packaging a proprietary Python library for multiple OSs Kevin Walzer <kw@codebykevin.com> - 2013-12-06 10:35 -0500

#61083 — Packaging a proprietary Python library for multiple OSs

Hi everyone,

I am developing a proprietary Python library. The library is currently Windows-only, and I want to also make it available for other platforms (Linux & Mac). I'm writing because I wanted to ask for your expert opinion on how to best do this.

The library is currently shipped in the form of a Zip file. This archive contains the compiled Python code for the implementation of my library, plus all dependencies. By placing the Zip file on his PYTHONPATH, the customer can use the library from his Python scripts.

Shipping a Zip file with all dependencies included has the following advantages:

 * No internet access or administrator privileges are required to install the library.
 * The customer does not have to worry about installing / managing dependencies of my library.

It also has the disadvantage that the customer is not (easily) able to use his own versions of my library's dependencies.

Even though I am not generating an EXE, I am using py2exe to obtain the distributable Zip file for my library. This "hack" is very convenient, as py2exe allows me to simply say which packages I require and does the work of performing a dependency analysis of the required libraries for me. py2exe automatically generates the Zip file with my (compiled) library code, and all dependencies.

Unfortunately, py2exe is only available for Windows. I need to also be able to build it on Linux & Mac, hence change the build process to not use py2exe.

My questions are:

 1. Is it considered a bad idea in the Python community to ship one large Zip file with all dependencies? From what I have seen, it seems to be an unusual approach, at the least. How do *you* prefer to obtain and install Python libraries?
 2. Is it possible to distribute the library in a form that allows for an offline installation without administrator privileges using other tools, such as setuptools?

My insight into the state of the art in Python regarding these matters is limited, so I would appreciate advice from someone with more experience in the subject.

A hard requirement is that I can only ship binary distributions of my library, as this is a proprietary product. I looked at Distutils and Setuptools, where the recommended approach seems to be to simply ship all sources.

Many thanks!

[toc] | [next] | [standalone]

#61084

On Thursday, December 5, 2013 3:44:50 PM UTC+5:30, Michael Herrmann wrote:
> Hi everyone,
>
> I am developing a proprietary Python library. The library is currently Windows-only, and I want to also make it available for other platforms (Linux & Mac). I'm writing because I wanted to ask for your expert opinion on how to best do this.

Wheel is the upcoming standard I think.
http://www.python.org/dev/peps/pep-0427/

1. It would be dishonest to remove the 'upcoming'
2. It would also be dishonest if you thought I know anything about the subject :-)
3. https://groups.google.com/forum/#!forum/python-virtualenv may be a better 
place to ask

[toc] | [prev] | [next] | [standalone]

#61091

On Dec 5, 2013, at 2:56 AM, rusi <rustompmody@gmail.com> wrote:

> 3. https://groups.google.com/forum/#!forum/python-virtualenv may be a better 
> place to ask

Am I the only one that sees the irony in this suggestion? Given the long running tirades^H^H^H^H^H^H thread about “Managing Google Groups headaches”?

“Pleassse don’t use Google Groupssssesss. It’sss nasssty. It hurtssess our eyesssessss with itsss long linessssieesss. Unlessssss it hassss a ssspecial neeeeed. Then the groupssesss are OK, Yessss?"

[toc] | [prev] | [next] | [standalone]

#61092

On Thursday, December 5, 2013 11:56:16 AM UTC+1, rusi wrote:
> Wheel is the upcoming standard I think.
> http://www.python.org/dev/peps/pep-0427/

I hadn't known of Wheel - thanks for pointing it out!

[toc] | [prev] | [next] | [standalone]

#61097

On Fri, Dec 6, 2013 at 2:32 AM, Travis Griggs <travisgriggs@gmail.com> wrote:
>
> On Dec 5, 2013, at 2:56 AM, rusi <rustompmody@gmail.com> wrote:
>
>> 3. https://groups.google.com/forum/#!forum/python-virtualenv may be a better
>> place to ask
>
> Am I the only one that sees the irony in this suggestion? Given the long running tirades^H^H^H^H^H^H thread about “Managing Google Groups headaches”?
>
> “Pleassse don’t use Google Groupssssesss. It’sss nasssty. It hurtssess our eyesssessss with itsss long linessssieesss. Unlessssss it hassss a ssspecial neeeeed. Then the groupssesss are OK, Yessss?"

No, it's not like that. It's that there are some people who, despite
truckloads of evidence to the contrary, still think that Google Groups
is worth using. Rusi is one of them. Fortunately, he has defended his
position by making his posts not look like the ridiculous junk that GG
creates by default, but that doesn't make GG a good product. It's like
an argument my boss and I had: I said that PHP is a bad language, and
he said that it can't possibly be a bad language because he's able to
write good code in it.

I don't know what mailing list there is for virtualenv as I don't use
it, but there's likely to be an alternative source of knowledge on it.

ChrisA

[toc] | [prev] | [next] | [standalone]

#61099

On Thursday, December 5, 2013 11:20:41 AM UTC-5, Chris Angelico wrote:

> No, it's not like that. It's that there are some people who, despite
> truckloads of evidence to the contrary, still think that Google Groups
> > is worth using. Rusi is one of them. Fortunately, he has defended his
> > position by making his posts not look like the ridiculous junk that GG
> > creates by default, but that doesn't make GG a good product.

I use GG on occasion (I'm using it now), when I don't have access to a better newsreader.  Like Rusi, I take the effort to clean up the double-space mess GG produces by default.  That doesn't mean GG isn't a piece of crap; it is.  That fact that I, and Rusi, know enough, and take the effort, to overcome its shortcomings doesn't change that.

I put GG it in the category of "attractive nuisance".  It's like leaving cans of spray paint laying around school playgrounds and then being surprised when the kids pick them up and use them to paint graffiti.  It certainly violates Google's "do no harm" motto when it interacts with usenet groups.

I keep hearing that I should use gmane as a superior interface.  Well, I tried that.  I went to http://dir.gmane.org/search.php, where it asks me to search for a newsgroup.  I type in "comp.lang.python", and it tells me, "No matching groups".  So, that seems pretty broken to me.

> It's like an argument my boss and I had: I said that PHP is a bad language, and
> he said that it can't possibly be a bad language because he's able to
> write good code in it.

PHP is a disaster of a language.  But, like any bad tool, a good craftsman can produce a quality product with it.  Wikipedia is written in PHP.  So, apparently, is gmane :-)  As much as I loathe working with PHP, I have to admit that if you can build a product like Wikipedia on it, it must have some redeeming qualities.

[toc] | [prev] | [next] | [standalone]

#61101

On Fri, Dec 6, 2013 at 4:12 AM, Roy Smith <roy@panix.com> wrote:
>> It's like an argument my boss and I had: I said that PHP is a bad language, and
>> he said that it can't possibly be a bad language because he's able to
>> write good code in it.
>
> PHP is a disaster of a language.  But, like any bad tool, a good craftsman can produce a quality product with it.  Wikipedia is written in PHP.  So, apparently, is gmane :-)  As much as I loathe working with PHP, I have to admit that if you can build a product like Wikipedia on it, it must have some redeeming qualities.

Right. And the fact that Wikipedia can be written in PHP is not itself
proof that it's a good language. You and Rusi are fighting against
GG's faults and not entirely succeeding, as your paragraphs come out
unwrapped; that's possibly the least of the GG woes, but it's one of
the first clues that someone's replies are likely to be double-spaced.

A good tool does most of your work for you. A bad tool has to be
fought every inch of the way. Sometimes a tool is good but wrongly
chosen (don't use DeScribe Macro Language for writing a GUI - drop to
REXX for that!), but some tools have no good use at all.

ChrisA

[toc] | [prev] | [next] | [standalone]

#61102

:

On Thu, Dec 05, 2013 at 09:12:30AM -0800, Roy Smith wrote:
> I keep hearing that I should use gmane as a superior interface.  Well,
> I tried that.  I went to http://dir.gmane.org/search.php, where it
> asks me to search for a newsgroup.  I type in "comp.lang.python", and
> it tells me, "No matching groups".  So, that seems pretty broken to
> me.

That's not entirely fair - Gmane presents mailing lists as newsgroups,
not vice versa, so it doesn't know that python-list@python.org is
connected to comp.lang.python (or that comp.lang.python even exists).

A search for the mailing list from the front page works just fine:

  http://gmane.org/find.php?list=python-list%40python.org

 -[]z.

-- 
Zero Piraeus: flagellum dei
http://etiol.net/pubkey.asc

[toc] | [prev] | [next] | [standalone]

#61103

On 05/12/2013 17:50, Zero Piraeus wrote:
> :
>
> On Thu, Dec 05, 2013 at 09:12:30AM -0800, Roy Smith wrote:
>> I keep hearing that I should use gmane as a superior interface.  Well,
>> I tried that.  I went to http://dir.gmane.org/search.php, where it
>> asks me to search for a newsgroup.  I type in "comp.lang.python", and
>> it tells me, "No matching groups".  So, that seems pretty broken to
>> me.
>
> That's not entirely fair - Gmane presents mailing lists as newsgroups,
> not vice versa, so it doesn't know that python-list@python.org is
> connected to comp.lang.python (or that comp.lang.python even exists).
>
> A search for the mailing list from the front page works just fine:
>
>    http://gmane.org/find.php?list=python-list%40python.org
>
>   -[]z.
>

Another useful link, there are just a few python goodies there 
http://news.gmane.org/index.php?prefix=gmane.comp.python

-- 
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#61104

On 2013-12-05 17:50, Zero Piraeus wrote:
> :
>
> On Thu, Dec 05, 2013 at 09:12:30AM -0800, Roy Smith wrote:
>> I keep hearing that I should use gmane as a superior interface.  Well,
>> I tried that.  I went to http://dir.gmane.org/search.php, where it
>> asks me to search for a newsgroup.  I type in "comp.lang.python", and
>> it tells me, "No matching groups".  So, that seems pretty broken to
>> me.
>
> That's not entirely fair - Gmane presents mailing lists as newsgroups,
> not vice versa, so it doesn't know that python-list@python.org is
> connected to comp.lang.python (or that comp.lang.python even exists).
>
> A search for the mailing list from the front page works just fine:
>
>    http://gmane.org/find.php?list=python-list%40python.org

Right. GMane is an NNTP service, but it is not part of the USENET network. 
comp.lang.python is a USENET newsgroup and requires a true USENET server (not 
just an NNTP server) to access.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#61100

On 05/12/2013 16:20, Chris Angelico wrote:
> On Fri, Dec 6, 2013 at 2:32 AM, Travis Griggs <travisgriggs@gmail.com> wrote:
>>
>> On Dec 5, 2013, at 2:56 AM, rusi <rustompmody@gmail.com> wrote:
>>
>>> 3. https://groups.google.com/forum/#!forum/python-virtualenv may be a better
>>> place to ask
>>
>> Am I the only one that sees the irony in this suggestion? Given the long running tirades^H^H^H^H^H^H thread about “Managing Google Groups headaches”?
>>
>> “Pleassse don’t use Google Groupssssesss. It’sss nasssty. It hurtssess our eyesssessss with itsss long linessssieesss. Unlessssss it hassss a ssspecial neeeeed. Then the groupssesss are OK, Yessss?"
>
> No, it's not like that. It's that there are some people who, despite
> truckloads of evidence to the contrary, still think that Google Groups
> is worth using. Rusi is one of them. Fortunately, he has defended his
> position by making his posts not look like the ridiculous junk that GG
> creates by default, but that doesn't make GG a good product. It's like
> an argument my boss and I had: I said that PHP is a bad language, and
> he said that it can't possibly be a bad language because he's able to
> write good code in it.
>
> I don't know what mailing list there is for virtualenv as I don't use
> it, but there's likely to be an alternative source of knowledge on it.
>
> ChrisA
>

gmane.comp.python.virtualenv

-- 
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#61106 — Official discussion forum for a project (was: Packaging a proprietary Python library for multiple OSs)

Travis Griggs <travisgriggs@gmail.com> writes:

> On Dec 5, 2013, at 2:56 AM, rusi <rustompmody@gmail.com> wrote:
>
> > 3. https://groups.google.com/forum/#!forum/python-virtualenv may be
> > a better place to ask
>
> Am I the only one that sees the irony in this suggestion? Given the
> long running tirades^H^H^H^H^H^H thread about “Managing Google Groups
> headaches”?

In addition to the fact that “rusi” evidently does not hold that
position, the Virtualenv project <URL:http://www.virtualenv.org/> has
its official discussion forum hosted at Google Groups, and the above
quote merely points that out.

One can recommend strongly against Google Groups in general as a
dreadful service, while still directing people to the official
discussion forum for a particular project which has made the (to
whatever degree misguided) decision to use that service.

-- 
 \         “Simplicity and elegance are unpopular because they require |
  `\           hard work and discipline to achieve and education to be |
_o__)                                appreciated.” —Edsger W. Dijkstra |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#61087

In article <58d49c5b-c837-4dac-b764-369fea02568c@googlegroups.com>,
 Michael Herrmann <michael.herrmann@heliumhq.com> wrote:

>  1. Is it considered a bad idea in the Python community to ship one large Zip 
>  file with all dependencies?

Yes.

> How do *you* prefer to obtain and install Python libraries?

"pip install"

>  2. Is it possible to distribute the library in a form that allows for an 
>  offline installation without administrator privileges using other tools, 
>  such as setuptools?

You can use "pip --find-links" to point pip at a local repository of 
packages.  That solves the offline part.  And the "without admin privs" 
part is solved by setting up a virtualenv.

> A hard requirement is that I can only ship binary distributions of my 
> library, as this is a proprietary product. I looked at Distutils and 
> Setuptools, where the recommended approach seems to be to simply ship all 
> sources.

Keep in mind that shipping just the pyc files offers very weak 
protection against people examining your code.  Google for "python 
decompile" and you'll find a number of projects.  I'm looking at the 
docs for uncompyle now, which says:

> 'uncompyle' converts Python byte-code back into equivalent Python
> source. It accepts byte-code from Python version 2.7 only.
> 
> The generated source is very readable: docstrings, lists, tuples and
> hashes get pretty-printed.

About the only thing not shipping Python source does is satisfy a 
check-box requirement that you not ship source.  It may make the lawyers 
and bean-counters happy, but that's about it.

[toc] | [prev] | [next] | [standalone]

#61093

On Thursday, December 5, 2013 3:09:32 PM UTC+1, Roy Smith wrote:
> >  1. Is it considered a bad idea in the Python community to ship one large Zip file with all dependencies?
> Yes.

I see. Unfortunately, the library's users may be non-technical and might not even have experience with Python. The easier the installation process, therefore, the better.

> > How do *you* prefer to obtain and install Python libraries?
> "pip install"

Thanks for this input.

> >  2. Is it possible to distribute the library in a form that allows for an 
> >  offline installation without administrator privileges using other tools, 
> >  such as setuptools?
> 
> You can use "pip --find-links" to point pip at a local repository of 
> packages.  That solves the offline part.  And the "without admin privs" 
> part is solved by setting up a virtualenv.

Both "pip --find-links" and "virtualenv" sound technically feasible but may be too difficult for my users (especially virtualenv).

> > A hard requirement is that I can only ship binary distributions of my 
> > library, as this is a proprietary product. I looked at Distutils and 
> > Setuptools, where the recommended approach seems to be to simply ship all 
> > sources.
> Keep in mind that shipping just the pyc files offers very weak 
> protection against people examining your code.  Google for "python 
> decompile" and you'll find a number of projects.  I'm looking at the 
> docs for uncompyle now, which says:
> > 'uncompyle' converts Python byte-code back into equivalent Python
> > source. It accepts byte-code from Python version 2.7 only.

Very interesting point. Thank you very much for pointing out uncompyle. I had always known that it was easy to decompile .pyc files, but hadn't imagined it to be that easy. I just tried uncompyle with some of our proprietary .pyc files. It took 5 minutes to set up and the results are near-perfect. Scary... :-S We might have to look into tools such as http://www.bitboost.com/#Python_obfuscator to obfuscate our code.

Thanks for the valuable insights!
Michael

[toc] | [prev] | [next] | [standalone]

#61095

On Thu, Dec 5, 2013, at 10:49, Michael Herrmann wrote:
> Very interesting point. Thank you very much for pointing out uncompyle. I
> had always known that it was easy to decompile .pyc files, but hadn't
> imagined it to be that easy. I just tried uncompyle with some of our
> proprietary .pyc files. It took 5 minutes to set up and the results are
> near-perfect. Scary... :-S We might have to look into tools such as
> http://www.bitboost.com/#Python_obfuscator to obfuscate our code.

Or you could just sue anyone who steals your code.

[toc] | [prev] | [next] | [standalone]

#61098

On Thursday, 5 December 2013 16:52:45 UTC+1, rand...@fastmail.us wrote:
> Or you could just sue anyone who steals your code.

I see your point but I don't think it's very practical. If the person who stole the code sits in some remote country with a completely different legal system, I think I'll have a hard time getting at this person. If I even manage to find out where the person is at all.

[toc] | [prev] | [next] | [standalone]

#61089

On 12/5/13, 5:14 AM, Michael Herrmann wrote:
> Even though I am not generating an EXE, I am using py2exe to obtain the distributable Zip file for my library. This "hack" is very convenient, as py2exe allows me to simply say which packages I require and does the work of performing a dependency analysis of the required libraries for me. py2exe automatically generates the Zip file with my (compiled) library code, and all dependencies.

If your library and their dependencies are simply .pyc files, then I 
don't see why a zip collated via py2exe wouldn't work on other 
platforms. Obviously this point is moot if your library includes true 
compiled (C-based) extensions.

--Kevin

-- 
Kevin Walzer
Code by Kevin/Mobile Code by Kevin
http://www.codebykevin.com
http://www.wtmobilesoftware.com

[toc] | [prev] | [next] | [standalone]

#61094

On Thursday, December 5, 2013 4:26:40 PM UTC+1, Kevin Walzer wrote:
> On 12/5/13, 5:14 AM, Michael Herrmann wrote:
> If your library and their dependencies are simply .pyc files, then I 
> don't see why a zip collated via py2exe wouldn't work on other 
> platforms. Obviously this point is moot if your library includes true 
> compiled (C-based) extensions.

As I said, I need to make my *build* platform-independent.

Thanks,
Michael

[toc] | [prev] | [next] | [standalone]

#61096

On 12/5/13, 10:50 AM, Michael Herrmann wrote:
> As I said, I need to make my *build* platform-independent.

cx_Freeze is platform independent, but I'm not sure if it generates 
libraries or simply executables.

--Kevin

-- 
Kevin Walzer
Code by Kevin/Mobile Code by Kevin
http://www.codebykevin.com
http://www.wtmobilesoftware.com

[toc] | [prev] | [next] | [standalone]

#61160

On 12/5/13, 10:50 AM, Michael Herrmann wrote:
> On Thursday, December 5, 2013 4:26:40 PM UTC+1, Kevin Walzer wrote:
>> On 12/5/13, 5:14 AM, Michael Herrmann wrote:
>> If your library and their dependencies are simply .pyc files, then I
>> don't see why a zip collated via py2exe wouldn't work on other
>> platforms. Obviously this point is moot if your library includes true
>> compiled (C-based) extensions.
>
> As I said, I need to make my *build* platform-independent.

Giving this further thought, I'm wondering how hard it would be to roll 
your own using modulefinder, Python's zip tools, and some custom code. 
Just sayin'.

--Kevin


-- 
Kevin Walzer
Code by Kevin/Mobile Code by Kevin
http://www.codebykevin.com
http://www.wtmobilesoftware.com

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web