Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.009 X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'tom': 0.07; 'builtin': 0.09; 'messing': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'subject:files': 0.09; 'python': 0.11; 'jan': 0.12; '65536': 0.16; 'archives?': 0.16; 'received:80.91.229.3': 0.16; 'received:plane.gmane.org': 0.16; 'reedy': 0.16; 'files.': 0.16; 'wrote:': 0.18; 'discussion': 0.18; 'looked': 0.18; 'trying': 0.19; 'import': 0.22; 'install': 0.23; 'header:User-Agent:1': 0.23; 'issue,': 0.24; 'header:X-Complaints- To:1': 0.27; 'header:In-Reply-To:1': 0.27; 'installed': 0.27; 'fixed': 0.29; 'url:bugs': 0.29; 'mode': 0.30; 'newer': 0.30; 'returned': 0.30; "i'm": 0.30; 'file': 0.32; 'run': 0.32; 'url:python': 0.33; 'limitation': 0.33; 'noticed': 0.34; 'could': 0.34; 'possible.': 0.35; 'but': 0.35; 'there': 0.35; 'version': 0.36; 'subject:?': 0.36; 'url:org': 0.36; 'should': 0.36; 'list': 0.37; 'to:addr:python-list': 0.38; 'files': 0.38; 'pm,': 0.38; 'anything': 0.39; 'to:addr:python.org': 0.39; 'mailing': 0.39; 'received:org': 0.40; 'skip:u 10': 0.60; 'read': 0.60; 'mentioned': 0.61; 'received:173': 0.61; 'simply': 0.61; 'box,': 0.64; 'search,': 0.74; 'click': 0.77; '3.4': 0.84; 'received:fios.verizon.net': 0.84; 'try.': 0.91; 'subject:limited': 0.93 X-Injected-Via-Gmane: http://gmane.org/ To: python-list@python.org From: Terry Reedy Subject: Re: zipimport limited to 65536 files? Date: Thu, 01 May 2014 19:03:24 -0400 References: <1398973761.73804.YahooMailNeo@web140106.mail.bf1.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Gmane-NNTP-Posting-Host: pool-173-75-254-207.phlapa.fios.verizon.net User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 In-Reply-To: <1398973761.73804.YahooMailNeo@web140106.mail.bf1.yahoo.com> X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 45 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1398985458 news.xs4all.nl 2975 [2001:888:2000:d::a6]:60851 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:70830 On 5/1/2014 3:49 PM, Tom Graves wrote: > Hello, > > I am trying to use python (2.6.6) to read a jar file that contains > python files. I'm simply setting > PYTHONPATH=spark-assembly-1.0.0-SNAPSHOT-hadoop2.4.0.jar. > Unfortunately it fails to read the python files from the jar file and > if run in verbose mode just shows: > > import zipimport # builtin > # installed zipimport hook > # zipimport: found 0 names in spark-assembly-1.0.0-SNAPSHOT-hadoop2.4.0.jar > > I was messing around and noticed that if I reduce the number of files > and directories in the jar to below 65536 then it works: > > import zipimport # builtin > # installed zipimport hook > # zipimport: found 65452 names in pyspark.jar > > Is this a known limitation It is definitely not documented, which it should be if intentional. > or is this perhaps fixed in newer version or Install 3.4 and try. Or go to http://bugs.python.org, click search, enter 'zipimport' in the title box, change 'open' in the status box to 'dont care', and look at 44 titles returned (I did not see anything that looked relevant). All I know is that in the discussion about a ziplib issue, someone mentioned using zips with 100000s of files. But zipimport could have an additional limitation. > is there a work around? Multiple archives? > Note, I'm not subscribed to the mailing list so please copy me in > response if possible. -- Terry Jan Reedy