Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!news.stack.nl!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python,': 0.02; '"this': 0.03; 'open-source': 0.04; '12px;': 0.05; 'classes,': 0.05; '(using': 0.07; 'distributing': 0.07; 'tries': 0.07; 'currently,': 0.09; 'cursor': 0.09; 'pointers': 0.09; 'skip:o 60': 0.09; 'tmp': 0.09; 'uses.': 0.09; 'python': 0.11; 'assume': 0.14; '__init__.py': 0.16; 'break-word;': 0.16; 'class),': 0.16; 'code?': 0.16; 'dimension': 0.16; 'distutils,': 0.16; 'example)': 0.16; 'module?': 0.16; 'non-python': 0.16; 'overriding': 0.16; 'permissions.': 0.16; 'space;': 0.16; 'subfolder': 0.16; 'subject:distribution': 0.16; 'subject:questions': 0.16; 'types,': 0.16; 'word-wrap:': 0.16; 'extensions': 0.16; 'helvetica;': 0.16; ':-)': 0.16; 'all.': 0.16; 'fix': 0.17; 'code.': 0.18; 'library': 0.18; 'obviously': 0.18; 'module': 0.19; 'file,': 0.19; 'thoughts': 0.19; 'examples': 0.20; 'written': 0.21; 'seems': 0.21; 'command': 0.22; 'code,': 0.22; 'install': 0.23; 'this?': 0.23; '': 0.91; 'canonical': 0.91 X-Warning: RFC compliance checks disabled due to whitelist X-Warning: Reverse-Path DNS check skipped due to whitelist X-Warning: Maximum message size check skipped due to whitelist X-Warning: Realtime Block Lists skipped due to whitelist X-Warning: System filters skipped due to whitelist X-Warning: Domain filters skipped due to whitelist X-Warning: User filters skipped due to whitelist X-Warning: Anti-Spam check skipped due to whitelist X-Whitelist: 2147483645 X-Envelope-From: israel@ravnalaska.net X-Envelope-To: python-list@python.org From: Israel Brewster Content-Type: multipart/alternative; boundary="Apple-Mail=_9F467185-2DE8-4BE3-B800-FF172D4DC888" Subject: CFFI distribution questions Date: Tue, 25 Nov 2014 08:53:44 -0900 To: python-list@python.org Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-Mailman-Approved-At: Tue, 25 Nov 2014 20:35:07 +0100 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 256 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1416944108 news.xs4all.nl 2830 [2001:888:2000:d::a6]:54734 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:81526 --Apple-Mail=_9F467185-2DE8-4BE3-B800-FF172D4DC888 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii So I have a python module that I have written which uses CFFI to link = against a C library I have compiled. Specifically, it is a Database = driver for the 4th dimension database, using an open-source C library = distributed by the 4D company. I have tested the module and C code on a = couple of different platforms, but I have a few questions regarding = where to go from here. 1) Currently, I am manually compiling the C library, and placing it in a = subfolder of the module. So the top level of the module directory = contains the python code and the __init__.py file, and then there is a = sub-directory (lib4d_sql) containing the C code and compiled C library. = I point CFFI to the library using a construct like this: _CWD =3D os.path.dirname(os.path.realpath(__file__)) ffi.verifier(..., Library_dirs=3D["{}/lib4d_sql/].format(_CWD), ...) Obviously I have left out a lot of code there. Is this sort of construct = considered kosher? Or is there a better way to say "this directory = relative to your location"? I can't just use relative paths, because = that would be relative to the execution directory (I think). 2) What is the proper way to compile and distribute the C library with = the python module?=20 The examples I've found about distributing a CFFI module all assume you = are are using some brain-dead-simple built-in C command where you don't = have to worry about compiling or installing a library. I found some = documentation related to building C and C++ extensions with distutils, = and following that managed to get the library to compile, but I can't = figure out what it does with the compiled library, or how to get it into = the proper location relative to my module. I also found some more "raw" = distutil code here: = https://coderwall.com/p/mjrepq/easy-static-shared-libraries-with-distutils= that I managed to use by overriding the finalize_options function of = the setup tools install class (using the cmdclass option of setup), and = this allowed me to build the library in the proper location in the tmp = install directory, but it doesn't seem to keep the library when = installing the module in the final location. So far, the only way I have = found to work around that is by including a "dummy" copy of the library = in the package_data option to setup, such that when the library is built = it replaces this dummy file. Is there a better/more canonical way to do = this? 3) The majority of the setup/packing procedure I pulled from here: = https://caremad.io/2014/11/distributing-a-cffi-project/ and it seems to = work - mostly. The one problem I am running into is with the implicit = compile that CFFI does. Some of the code (most, really) on that page is = designed to work around this by doing the compile on install so it = doesn't have to be done at runtime, however this doesn't appear to be = working for me. I see signs that it is doing the compile at install = time, however when I try to run the module it still tries to compile at = that time, an operation that fails due to lack of permissions. Might = anyone have some thoughts on how I can fix this? Finally, let me ask you guys this: is distributing this as a CFFI module = even the right approach? Or should I be looking at something else = entirely? C code isn't needed for speed - the database itself will be = way slower than the interface code. It's just that the driver is = distributed as C code and I didn't want to have to reverse engineer = their code and re-write it all. I've read over = https://docs.python.org/2/extending/extending.html, but I think I'm = missing something when it comes to actually interfacing with python, and = dealing with the non-python types (pointers to a struct, for example) = that the C library uses. Would I have to put 100% of my code, including = things like the cursor and connection classes, in C code? When I call = the connect function from python (which should return an instance of a = connection class), what exactly should my C function return? All the = examples on that page show basic C types, not pointers to class = instances or the like. Maybe I just need to read over the documentation = a few more times :-) Thanks for any help anyone can provide on any of these questions! :-) = Sorry for being so long-winded. ----------------------------------------------- Israel Brewster Systems Analyst II Ravn Alaska 5245 Airport Industrial Rd Fairbanks, AK 99709 (907) 450-7293 ----------------------------------------------- --Apple-Mail=_9F467185-2DE8-4BE3-B800-FF172D4DC888 Content-Type: multipart/mixed; boundary="Apple-Mail=_C8A5CE93-28DD-4F6F-937F-21C3B56492F6" --Apple-Mail=_C8A5CE93-28DD-4F6F-937F-21C3B56492F6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii So I = have a python module that I have written which uses CFFI to link against = a C library I have compiled. Specifically, it is a Database driver for = the 4th dimension database, using an open-source C library distributed = by the 4D company. I have tested the module and C code on a couple of = different platforms, but I have a few questions regarding where to go = from here.

1) Currently, I am manually compiling the = C library, and placing it in a subfolder of the module. So the top level = of the module directory contains the python code and the __init__.py = file, and then there is a sub-directory (lib4d_sql) containing the C = code and compiled C library. I point CFFI to the library using a = construct like this:

_CWD =3D = os.path.dirname(os.path.realpath(__file__))
ffi.verifier(..., = Library_dirs=3D["{}/lib4d_sql/].format(_CWD), = ...)

Obviously I have left out a lot of code = there. Is this sort of construct considered kosher? Or is there a better = way to say "this directory relative to your location"? I can't just use = relative paths, because that would be relative to the execution = directory (I think).

2) What is the proper way = to compile and distribute the C library with the python = module? 

The examples I've found about = distributing a CFFI module all assume you are are using some = brain-dead-simple built-in C command where you don't have to worry about = compiling or installing a library. I found some documentation related to = building C and C++ extensions with distutils, and following that managed = to get the library to compile, but I can't figure out what it does with = the compiled library, or how to get it into the proper location relative = to my module. I also found some more "raw" distutil code here: https://coderwall.com/p/mjrepq/easy-static-shared-libraries-with= -distutils that I managed to use by overriding the finalize_options = function of the setup tools install class (using the cmdclass option of = setup), and this allowed me to build the library in the proper location = in the tmp install directory, but it doesn't seem to keep the library = when installing the module in the final location. So far, the only way I = have found to work around that is by including a "dummy" copy of the = library in the package_data option to setup, such that when the library = is built it replaces this dummy file. Is there a better/more canonical = way to do this?

3) The majority of the = setup/packing procedure I pulled from here: https://c= aremad.io/2014/11/distributing-a-cffi-project/ and it seems to work = - mostly. The one problem I am running into is with the implicit compile = that CFFI does. Some of the code (most, really) on that page is designed = to work around this by doing the compile on install so it doesn't have = to be done at runtime, however this doesn't appear to be working for me. = I see signs that it is doing the compile at install time, however when I = try to run the module it still tries to compile at that time, an = operation that fails due to lack of permissions. Might anyone have some = thoughts on how I can fix this?

Finally, let me = ask you guys this: is distributing this as a CFFI module even the right = approach? Or should I be looking at something else entirely? C code = isn't needed for speed - the database itself will be way slower than the = interface code. It's just that the driver is distributed as C code and I = didn't want to have to reverse engineer their code and re-write it all. = I've read over https://docs.p= ython.org/2/extending/extending.html, but I think I'm missing = something when it comes to actually interfacing with python, and dealing = with the non-python types (pointers to a struct, for example) that the C = library uses. Would I have to put 100% of my code, including things like = the cursor and connection classes, in C code? When I call the connect = function from python (which should return an instance of a connection = class), what exactly should my C function return? All the examples on = that page show basic C types, not pointers to class instances or the = like. Maybe I just need to read over the documentation a few more times = :-)

Thanks for any help anyone can provide on = any of these questions! :-) Sorry for being so = long-winded.


-----------------------------------------------
Israel = Brewster
Systems = Analyst II
Ravn = Alaska
5245 = Airport Industrial Rd
Fairbanks, AK = 99709
(907) = 450-7293
-----------------------------------------------
<= div style=3D"margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: = 'Times New Roman', serif;">

= --Apple-Mail=_C8A5CE93-28DD-4F6F-937F-21C3B56492F6 Content-Disposition: attachment; filename="Israel Brewster.vcf" Content-Type: text/directory; name="Israel Brewster.vcf" Content-Transfer-Encoding: quoted-printable BEGIN:VCARD=0D=0AVERSION:3.0=0D=0AN:Brewster;Israel;;;=0D=0AFN:Israel=20= Brewster=0D=0AORG:Frontier=20Flying=20Service;MIS=0D=0ATITLE:PC=20= Support=20Tech=20II=0D=0A= EMAIL;type=3DINTERNET;type=3DWORK;type=3Dpref:israel@frontierflying.com=0D= =0ATEL;type=3DWORK;type=3Dpref:907-450-7293=0D=0A= item1.ADR;type=3DWORK;type=3Dpref:;;5245=20Airport=20Industrial=20= Wy;Fairbanks;AK;99701;=0D=0Aitem1.X-ABADR:us=0D=0ACATEGORIES:General=0D=0A= X-ABUID:36305438-95EA-4410-91AB-45D16CABCDDC\:ABPerson=0D=0AEND:VCARD=0D=0A= --Apple-Mail=_C8A5CE93-28DD-4F6F-937F-21C3B56492F6 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=us-ascii

--Apple-Mail=_C8A5CE93-28DD-4F6F-937F-21C3B56492F6-- --Apple-Mail=_9F467185-2DE8-4BE3-B800-FF172D4DC888--