Path: csiph.com!usenet.pasdenom.info!news.albasani.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!194.109.133.85.MISMATCH!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <python.list@tim.thechases.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.002
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'debugging': 0.05; 'append': 0.07; 'python': 0.09; 'cc:addr:python-list': 0.10; '-tkc': 0.16; '2.7.3': 0.16; 'dump': 0.16; 'extensions,': 0.16; 'from:addr:python.list': 0.16; 'from:addr:tim.thechases.com': 0.16; 'from:name:tim chase': 0.16; 'imap4': 0.16; 'message- id:@tim.thechases.com': 0.16; 'perl.': 0.16; 'received:70.251': 0.16; 'received:dsl.rcsntx.swbell.net': 0.16; 'received:rcsntx.swbell.net': 0.16; 'received:swbell.net': 0.16; 'seconds.': 0.16; 'subject:slow': 0.16; 'wrote:': 0.17; 'issue,': 0.17; 'code,': 0.18; 'code.': 0.20; 'cc:2**0': 0.23; 'work.': 0.23; 'cc:no real name:2**0': 0.24; 'tried': 0.25; 'cc:addr:python.org': 0.25; 'header:In-Reply-To:1': 0.25; 'header :User-Agent:1': 0.26; 'looks': 0.26; '[1]': 0.27; 'noticed': 0.28; 'appending': 0.29; 'bad.': 0.29; 'cpu': 0.29; 'i/o': 0.29; 'perl': 0.29; 'way?': 0.29; "i'm": 0.29; 'seconds': 0.30; 'code': 0.31; 'problem.': 0.32; '5000': 0.33; 'done': 0.34; 'server': 0.35; 'doing': 0.35; 'there': 0.35; 'url:org': 0.36; 'does': 0.37; 'being': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'files': 0.38; 'some': 0.38; 'page': 0.38; 'performance': 0.39; 'takes': 0.39; 'procedure': 0.61; 'profile': 0.61; 'results': 0.65; 'taking': 0.65; 'reply': 0.66; 'inbox.': 0.78; 'ins': 0.84; 'received:50.22': 0.84; 'subject:very': 0.91
Date: Wed, 25 Jul 2012 22:21:25 -0500
From: Tim Chase <python.list@tim.thechases.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111120 Icedove/3.1.16
MIME-Version: 1.0
To: Simon Pirschel <sp@abusix.org>
Subject: Re: append in IMAP4 from imaplib very slow
References: <5010314F.80001@abusix.com>
In-Reply-To: <5010314F.80001@abusix.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - boston.accountservergroup.com
X-AntiAbuse: Original Domain - python.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - tim.thechases.com
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Cc: python-list@python.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.2584.1343272821.4697.python-list@python.org>
Lines: 32
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1343272821 news.xs4all.nl 6990 [2001:888:2000:d::a6]:52603
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:26054

On 07/25/12 12:47, Simon Pirschel wrote:
> I'm currently experimenting with IMAP using Python 2.7.3 and
> IMAP4 from imaplib. I noticed the performance to be very bad. I
> read 5000 files from a directory and append them to an IMAP
> INBOX. The hole procedure of reading and appending is taking
> about 210 seconds.
> 
> I set up the exact same code in Perl to check if there is a
> general IMAP server configuration issue, since CPU and I/O isn't
> the problem. The same amount of data on the same IMAP server is
> done after 7.9 seconds using Perl.
> 
> The difference is huge and I tried to narrow the issue down by
> profiling the Python code. The profile results are, 206 seconds
> are spent in calling socket.recv.

While I don't know the ins and outs of the Perl code, it looks like
the Python imaplib.py is doing some pretty straight-forward
send/receive work.  However, IMAP4 does support both pipelining and
multi-append extensions, so if the Perl code takes advantage of the
features if the server offers them, it might be reaping significant
gains over the naïve Python code.

Can you get a debugging dump of the commands being sent each way?
(particularly if you see a bunch of APPEND commands before the reply
acknowledgement, as detailed at [1])

-tkc

[1]
http://www.ietf.org/rfc/rfc3502.txt on page 4