Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail From: Francesc Alted Newsgroups: comp.lang.python.announce Subject: ANN: bcolz 1.1.0 released! Date: Fri, 10 Jun 2016 17:28:09 +0200 Lines: 122 Approved: python-announce-list@python.org Message-ID: References: Reply-To: faltet@gmail.com Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Trace: news.uni-berlin.de voq+DeIM5vO2jd5tXRm8bA3snWYbRdEbkX+3FOTXb/BQ== Return-Path: X-Original-To: python-announce-list@python.org Delivered-To: python-announce-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'binary': 0.05; 'defaults': 0.05; 'repository': 0.05; '(so': 0.07; 'column': 0.07; 'expressions': 0.07; 'subject:ANN': 0.07; 'subject:released': 0.07; 'below).': 0.09; 'cc:addr:googlegroups.com': 0.09; 'compression': 0.09; 'fine-tune': 0.09; 'git': 0.09; 'internally': 0.09; 'moreover,': 0.09; 'rows': 0.09; 'url:github': 0.09; 'url:rst': 0.09; 'systems.': 0.11; 'url:)': 0.14; '(),': 0.16; '1.1.0': 0.16; '255': 0.16; '==========': 0.16; 'alted': 0.16; 'columns': 0.16; 'containers': 0.16; 'efficiently.': 0.16; 'evaluating': 0.16; 'francesc': 0.16; 'from:addr:faltet': 0.16; 'from:name:francesc alted': 0.16; 'iterators': 0.16; 'now.\xc2\xa0': 0.16; 'numpy': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'too).': 0.16; 'url:licenses': 0.16; 'url:talks': 0.16; 'app': 0.16; 'memory': 0.17; 'cheap': 0.18; 'windows': 0.20; 'cc:addr:python.org': 0.20; '(see': 0.20; 'machine': 0.21; 'cc:2**1': 0.22; '64-bit': 0.22; 'arrays': 0.22; 'see:': 0.22; 'skip:= 20': 0.22; "user's": 0.22; 'cc:no real name:2**0': 0.22; '(like': 0.23; 'performing': 0.23; 'specially': 0.23; 'tables': 0.23; 'unix': 0.24; 'example': 0.26; 'addition,': 0.27; '----': 0.27; 'least': 0.27; 'message-id:@mail.gmail.com': 0.27; '(although': 0.29; '32-bit': 0.29; 'closer': 0.29; 'comparison': 0.29; 'fast.': 0.29; 'gil': 0.29; 'i/o': 0.29; 'objects': 0.29; 'tutorial': 0.29; 'typically': 0.29; 'allows': 0.30; 'minimal': 0.30; 'query': 0.30; 'operations': 0.31; 'at:': 0.31; 'another': 0.32; 'implement': 0.32; 'getting': 0.33; 'point': 0.33; 'platforms.': 0.33; 'received:google.com': 0.35; 'filter': 0.35; 'i.e.': 0.35; 'list:': 0.35; 'quite': 0.35; 'supports': 0.35; 'url:org': 0.36; 'received:209.85': 0.36; 'possible': 0.36; 'data.': 0.36; 'faster': 0.36; 'framework': 0.36; 'notes': 0.36; 'subject:: ': 0.37; 'method': 0.37; 'operating': 0.37; 'thanks': 0.37; 'release': 0.37; 'doing': 0.38; 'creation': 0.38; 'virtual': 0.38; 'received:209': 0.38; 'several': 0.38; 'test': 0.39; 'data': 0.39; 'format': 0.39; 'does': 0.39; 'resources': 0.39; 'called': 0.40; 'some': 0.40; 'ever': 0.60; 'url:group': 0.61; 'default': 0.61; 'real': 0.62; 'improved': 0.63; 'more': 0.63; 'visit': 0.64; 'between': 0.65; 'license': 0.65; 'fundamental': 0.66; 'promise': 0.66; 'header :Reply-To:1': 0.67; 'home': 0.67; 'finally': 0.70; 'reply-to:no real name:2**0': 0.71; 'special': 0.73; 'reply-to:addr:gmail.com': 0.76; 'carried': 0.76; 'removal': 0.79; 'column.': 0.84; 'dimensional': 0.84; 'faster.': 0.84; 'highlights:': 0.84; 'lastly,': 0.84; 'together,': 0.84; 'url:master': 0.84; 'announcing': 0.91; 'chapters': 0.93; 'engines.': 0.93; 'reducing': 0.93; 'email addr:googlegroups.com': 0.95 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:from:date:message-id:subject:to:cc; bh=cylg805EswTx6yedEu2fTfshqZIZJ1uidrqG7An3NY0=; b=TZ2V5j6yq0LOdADNUdo+4u+Y00qW6/yu/hfTyLVCeHxgMgvtZOWtkF2acO4xnQFuDP kJ5JsMTAnsR7ZXYobwSGp48OIPtASbGpUsgKoZuUfjhVNKmOD4OKL4S4o6eq5Hrflbds prAiXH2H26Ud+gycrAx1FUGQbRC9J3t4XLZwVS9/O+n9OIyI0KKiWXwt+8m7Fi1NCyd4 D+mi+7RLvkKda1dwtmlDHT2b+Bei9V51v6MyX1yNe7B0aTQBZ+1o9G5esgj2LXxfgpCU xmc2aZgy4+GIL5pWT0lkKuQuV2n3yUCJTH1l5+rOvyG81Vgr7TSeZRmNEf8siXZ/tTeR zoSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:reply-to:from:date:message-id :subject:to:cc; bh=cylg805EswTx6yedEu2fTfshqZIZJ1uidrqG7An3NY0=; b=jheVztEQ18I/zRFM09Qdm6Gst2ppaX4xIPE9kVJUzSoiDXM6PQgF/4mb9OhE1jwNkp IVMTssqDharjxzGM/UxoyOYywZ2exyph9dKgoAO5eBlHukzld1t/VaqATYy5EwiG6lZy DiCEVPJqe9bTRwuukcqDIVDZHyaaShqkxGb1ifQ+rsOKz4uLUL8Z2L+sPjx2QzOgKbhB ZaVg9Tuxo8DrAJz9rkGXuiDZpaKhuqS+wpBDArxG38REVRxnrZlPjMKnkvi7X+cQfFiL O+Ii2ruDp714bwDgWJhaNxPKuPwerarGOTBd1fS3ypBO3aI0MxX3HocJ6br9mFXWK3qo 7N0w== X-Gm-Message-State: ALyK8tJu+XjIgcgKfFICy4ZEz/5OrtjB16/14CoK0qEDle6IGP3z4dLU81XzSRolRWUaMevdEZNuRB2sLIUMaQ== X-Received: by 10.13.227.69 with SMTP id m66mr1384641ywe.302.1465572490017; Fri, 10 Jun 2016 08:28:10 -0700 (PDT) X-Mailman-Approved-At: Fri, 10 Jun 2016 11:29:21 -0400 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: python-announce-list@python.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Announcement-only list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Mailman-Original-Message-ID: Xref: csiph.com comp.lang.python.announce:2174 ====================== Announcing bcolz 1.1.0 ====================== What's new ========== This release brings quite a lot of changes. After format stabilization in 1.0, the focus is now in fine-tune many operations (specially queries in ctables), as well as widening the available computational engines. Highlights: * Much improved performance of ctable.where() and ctable.whereblocks(). Now bcolz is getting closer than ever to fundamental memory limits during queries (see the updated benchmarks in the data containers tutorial below). * Better support for Dask; i.e. GIL is released during Blosc operation when bcolz is called from a multithreaded app (like Dask). Also, Dask can be used as another virtual machine for evaluating expressions (so now it is possible to use it during queries too). * New ctable.fetchwhere() method for getting the rows fulfilling some condition in one go. * New quantize filter for allowing lossy compression of floating point data. * It is possible to create ctables with more than 255 columns now. Thanks to Skipper Seabold. * The defaults during carray creation are scalars now. That allows to create highly dimensional data containers more efficiently. * carray object does implement the __array__() special method now. With this, interoperability with numpy arrays is easier and faster. For a more detailed change log, see: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst For some comparison between bcolz and other compressed data containers, see: https://github.com/FrancescAlted/DataContainersTutorials specially chapters 3 (in-memory containers) and 4 (on-disk containers). What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/querying-ep14.ipynb#Plots Example users of bcolz are Visualfabriq (http://www.visualfabriq.com/), and Quantopian (https://www.quantopian.com/): * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz@googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!** -- Francesc Alted