Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python,': 0.02; 'diff': 0.05; 'failing': 0.05; 'lines,': 0.05; 'badly': 0.07; 'doctest': 0.07; 'encoder': 0.07; 'json': 0.07; 'dict': 0.09; 'instance.': 0.09; 'output,': 0.09; 'repr': 0.09; 'ruby,': 0.09; 'cc:addr :python-list': 0.10; 'output': 0.15; 'thu,': 0.15; 'doctests': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'keys.': 0.16; 'sorting': 0.16; 'splits': 0.16; 'subject:make': 0.16; 'textual': 0.16; 'url:%3e': 0.16; 'url:json': 0.16; 'url:standards': 0.16; "{'a':": 0.16; 'wrote:': 0.16; 'module,': 0.18; 'examples': 0.18; 'tests': 0.18; 'library': 0.20; 'skip:" 30': 0.20; 'cc:2**0': 0.21; 'cc:addr:python.org': 0.21; 'keys': 0.22; 'subject:request': 0.22; 'pass': 0.22; 'am,': 0.23; '2015': 0.23; 'module': 0.23; "python's": 0.23; 'file.': 0.24; 'header:In- Reply-To:1': 0.24; 'somewhere': 0.24; 'separate': 0.27; '[2]': 0.27; 'message-id:@mail.gmail.com': 0.28; 'subject:/': 0.29; 'buffers': 0.29; 'such.': 0.29; 'url:se': 0.29; 'e.g.': 0.31; 'option': 0.31; '[1]': 0.32; 'url:python': 0.33; 'problem': 0.33; 'equal': 0.34; 'received:google.com': 0.34; 'useful': 0.35; 'involving': 0.35; 'protocol': 0.35; 'but': 0.36; 'text': 0.36; 'url:org': 0.36; 'there': 0.36; 'url:library': 0.36; 'two': 0.37; 'should': 0.37; "didn't": 0.37; 'subject:: ': 0.37; 'rather': 0.38; 'url:docs': 0.39; 'where': 0.40; 'why': 0.40; 'easy': 0.60; 'url:3': 0.60; 'your': 0.60; 'even': 0.61; 'different': 0.64; 'url:0': 0.66; 'worth': 0.73; 'url:search': 0.83; 'chrisa': 0.84; 'coupled': 0.84; 'subject:read': 0.84; 'subject:write': 0.84; 'to:none': 0.90; 'canonical': 0.91 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=bpu6CRF3qdcxVQ0q2TCs+KQ3rcIwgIoDtmUChPkqqt0=; b=OdCS+EUDzZAKq15WyN8GoTrNzlDMrbWzuG2IZobWc1BCJspq3fx5EXmMSKJsVtoRHN w86aw8Z2PZEK2G3MnlEwLUxHNipGPE6jSOWNDNdusp49Wd19Yhy4eR+kPChr690bdT1x 68sOObXOO/NJ98fYUHoNpS0vB0oItggwm26iJE+0RLrG+YcqNEHtw/Pann/kGcVPoYf2 ivWadWVqLwlT5wC2wqoLeCn4hmOA2AH/LYnpdNDnWnVQUG4jPau1BU8nAXL4Y/oDeUCH lZCuyRvRsoX8mJBWoC4oEhAXjw7UKeCA3/rZu5F+Y3g2DXF/KLY3iz0cxbAsvZ5adUWq yLhQ== MIME-Version: 1.0 X-Received: by 10.50.176.228 with SMTP id cl4mr9327657igc.2.1433980688969; Wed, 10 Jun 2015 16:58:08 -0700 (PDT) In-Reply-To: References: <55778208$0$2899$e4fe514c@news2.news.xs4all.nl> <5577b46d$0$12975$c3e8da3$5496439d@news.astraweb.com> <5578053a$0$11102$c3e8da3@news.astraweb.com> Date: Thu, 11 Jun 2015 09:58:08 +1000 Subject: Re: enhancement request: make py3 read/write py2 pickle format From: Chris Angelico Cc: "comp.lang.python" Content-Type: text/plain; charset=UTF-8 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.20+ Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 34 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1433980692 news.xs4all.nl 2961 [2001:888:2000:d::a6]:53550 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:92435 On Thu, Jun 11, 2015 at 8:10 AM, Devin Jeanpierre wrote: > The problem is that there are two different ways repr might write out > a dict equal to {'a': 1, 'b': 2}. This can make tests brittle -- e.g. > it's why doctest fails badly at examples involving dictionaries. Text > format protocol buffers output everything sorted, so that you can do > textual diffs for compatibility tests and such. With Python's JSON module [1], you can pass sort_keys=True to stipulate that the keys be lexically ordered, which should make the output "canonical". Pike's Standards.JSON.encode() [2] can take a flag value to canonicalize the output, which currently has the same effect (sort mappings by their indices). I did a quick check for Ruby and didn't find anything in its standard library JSON module, but knowing Ruby, it'll be available somewhere in a gem. A web search for 'perl json' brought up a CPAN link [4] that has a canonicalize option for sorting by keys. So that's three out of four definite, one uncertain, where it's pretty easy to ensure that you get byte-for-byte identical output from a JSON encoder. Even though failing doctests are a separate problem, it's useful to have canonical output. Your diffs get less noisy, for instance. Coupled with a human-readability flag (eg "indent=4" in Python, "Standards.JSON.HUMAN_READABLE" in Pike) that splits the result over multiple lines, it can make a pretty easy to diff file. Definitely worth doing... and definitely worth using a JSON encoder rather than repr(). ChrisA [1] https://docs.python.org/3/library/json.html#json.dump [2] http://pike.lysator.liu.se/generated/manual/modref/ex/predef_3A_3A/Standards/JSON.html [3] http://ruby-doc.org/stdlib-2.0.0/libdoc/json/rdoc/JSON.html [4] http://search.cpan.org/~makamaka/JSON-2.90/lib/JSON.pm#PERL_-%3E_JSON