Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail From: Chris Angelico Newsgroups: comp.lang.python Subject: Re: intermittent ValueErrors from subprocess Date: Mon, 16 May 2016 09:32:37 +1000 Lines: 39 Message-ID: References: <57389A9D.1010205@gmail.com> <5738F8E2.1030403@gmail.com> <57390395.8070005@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Trace: news.uni-berlin.de RHYOfx9oJ633eQygKxoXNAQUTQHPUMuXRRyHHh6aX4jA== Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; '16,': 0.03; 'that?': 0.05; 'apis': 0.07; 'executed': 0.07; 'means,': 0.07; 'api': 0.09; 'cc:addr:python-list': 0.09; '"run': 0.09; 'exits': 0.09; 'mess': 0.09; 'messing': 0.09; 'yeah,': 0.09; 'exception': 0.13; '12:59': 0.16; '2016': 0.16; "can't.": 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'pipes': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'so;': 0.16; 'traceback.': 0.16; 'zero,': 0.16; 'wrote:': 0.16; 'cc:2**0': 0.20; 'cc:addr:python.org': 0.20; 'fix': 0.21; 'explicit': 0.22; 'simpler': 0.22; 'trace': 0.22; 'am,': 0.23; 'slightly': 0.23; 'somewhere': 0.24; 'header:In-Reply-To:1': 0.24; 'mon,': 0.24; 'chris': 0.26; 'least': 0.27; 'message-id:@mail.gmail.com': 0.27; 'occurred': 0.29; 'such.': 0.29; 'program,': 0.29; 'code': 0.30; "i'd": 0.31; 'post': 0.31; "can't": 0.32; 'says': 0.32; 'problem': 0.33; 'source': 0.33; 'usually': 0.33; 'case,': 0.34; 'received:google.com': 0.35; 'could': 0.35; 'instance': 0.35; 'something': 0.35; 'level': 0.35; 'received:74.125.82': 0.35; 'problem.': 0.35; 'but': 0.36; 'lines': 0.36; 'smaller': 0.36; 'subject:: ': 0.37; 'being': 0.37; 'there,': 0.37; 'client': 0.37; 'say': 0.37; 'times.': 0.38; 'several': 0.38; 'someone': 0.38; 'mean': 0.38; 'sure': 0.39; 'subject:from': 0.39; "didn't": 0.39; 'unable': 0.39; 'rather': 0.39; 'high': 0.60; 'your': 0.60; 'clients': 0.61; 'worth': 0.67; 'apart': 0.70; 'click': 0.76; 'demand': 0.79; 'chrisa': 0.84; "else's": 0.84; 'hard.': 0.84; 'received:74.125.82.47': 0.84; 'start.': 0.84; 'tease': 0.84; 'to:none': 0.91 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-transfer-encoding; bh=VHp+K0j62uYew3IZBFydfSqHrp3qeQsxgshewmN8VnI=; b=RlbAuPgLpAtp+4OTqsVJzi3bQ1V1zH/UVVWcH1PZW0ERJ8xKFHYey6I2HdbcjtKOlH ACDAPjCNwqWbk4pAEsCANlPBHi8VGtxKrXbkoeW5Wk/Dh26RgdZB1BNeW3w4wIXFwLv3 keErnZO3A+vVC1fB5/Rm7znJG7jzb9yVX+wgVvMwbf3oxV54sw838fDhZVqf9oqauoPH JOUUVlrn02qbYbZYTFRtZPCjUq+SNqnl/JrxMyNrW+JTnM99gMZGY/k55FmiOnISHSS+ xzMWtzcekIVosgy/dGxxc0+4AKxoltMskfYQLMOcHRGNXEwIaMr8WJGJb3LHr6/cq0dz UHOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:cc:content-transfer-encoding; bh=VHp+K0j62uYew3IZBFydfSqHrp3qeQsxgshewmN8VnI=; b=kk+wOHKSLMxy+cUppZJufLyV7udYIKeFnBGUIxPE359UssMkfjRJX+8X3vFqyNiyNJ drJRC4nmDVjsDXpNLsRfrHhpeEuo81irdHgcuK2BOy8Hi6ErBoplE8JhSsW0bU7eu+TP qvt7qRUngffjhvzRVO0xAaOOkdi/cfL9YWthL+tD074qs47aMpVAhkY/CgtNZ9bTRueN x1IEC2Lxybp7LS9Qgk7rkH9yfAzDZ4F/GnEk3GhscJxilchnqcQjdCY8qJk9YdhY8Kmx zrw4mfnmW+f8wIQAMsXvmj51mlzoyzWUJagDMGtJiKsoeC6wadLn3d4u0MlhqPxHmWIc YlDA== X-Gm-Message-State: AOPr4FUsPKjKsJj3/5alqB32Mg+W3/76nrJogivgheC8AadvXLiPCzljsdSFZmyIxxFHRja8j8NeiAl8KzGrew== X-Received: by 10.194.233.97 with SMTP id tv1mr13059417wjc.30.1463355157784; Sun, 15 May 2016 16:32:37 -0700 (PDT) In-Reply-To: <57390395.8070005@gmail.com> X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Mailman-Original-Message-ID: X-Mailman-Original-References: <57389A9D.1010205@gmail.com> <5738F8E2.1030403@gmail.com> <57390395.8070005@gmail.com> Xref: csiph.com comp.lang.python:108651 On Mon, May 16, 2016 at 9:17 AM, Pavlos Parissis wrote: > On 16/05/2016 12:59 =CF=80=CE=BC, Chris Angelico wrote: >> Ah. I didn't click that link in your original post - didn't know it >> was the traceback. Better would have been to at least say so; best >> would have been to include it inline. >> > > I don't usually include traces as they have long lines and several > e-mail clients mess with them in a way that makes the trace unreadable. If your client can't send them, fix your client. If someone else's client can't read them, that's not your problem. >> If you can recreate the problem consistently, > > I can't. This is my main problem. This code has been executed ~100K and > that exception has occurred only ~5 times. Yeah, I figured. Makes it tough. >> it would be worth >> messing around with slightly lower level APIs - using subprocess.Popen >> rather than check_output, for instance - and see what you can do >> without the pipes. > > What do you mean by that? You're currently using a very high level API that says "run this program, make sure it exits zero, and give me its output". Somewhere inside there, something's going wrong. So the first thing I'd do would be to tease apart the job into smaller and simpler parts; that means, in this case, going to something like subprocess.Popen, with explicit use of pipes and such. You could look at the source code for check_output for a start. But being unable to recreate the problem on demand makes it hard. ChrisA