Path: csiph.com!feeder.erje.net!2.eu.feeder.erje.net!newsfeed.kamp.net!newsfeed.kamp.net!fu-berlin.de!uni-berlin.de!not-for-mail From: Francesc Alted Newsgroups: comp.lang.python.announce Subject: [ANN] bcolz 1.0.0 RC1 released Date: Tue, 8 Mar 2016 14:27:44 +0100 Lines: 129 Approved: python-announce-list@python.org Message-ID: Reply-To: faltet@gmail.com Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Trace: news.uni-berlin.de RYlbE6tM7P3bWBwCVRmS5wNA8kOAKZv1/qEdihLk/61A== Return-Path: X-Original-To: python-announce-list@python.org Delivered-To: python-announce-list@mail.python.org X-Spam-Status: OK 0.003 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'binary': 0.05; 'repository': 0.05; 'column': 0.07; 'implements': 0.07; 'pypi': 0.07; 'subject:ANN': 0.07; 'subject:released': 0.07; '1.0.0': 0.09; 'cc:addr:googlegroups.com': 0.09; 'compression': 0.09; 'git': 0.09; 'internally': 0.09; 'moreover,': 0.09; 'url:github': 0.09; 'url:rst': 0.09; 'yeah,': 0.09; 'bug': 0.10; 'systems.': 0.11; 'url:)': 0.14; 'backend': 0.15; '(),': 0.16; '(more': 0.16; '1.x': 0.16; '==========': 0.16; 'alted': 0.16; 'containers': 0.16; 'filters,': 0.16; 'francesc': 0.16; 'from:addr:faltet': 0.16; 'from:name:francesc alted': 0.16; 'iterators': 0.16; 'numpy': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'too).': 0.16; 'url:licenses': 0.16; 'url:talks': 0.16; 'memory': 0.17; 'cheap': 0.18; 'variable': 0.18; 'subject:] ': 0.19; 'windows': 0.20; 'cc:addr:python.org': 0.20; 'cc:2**2': 0.20; '2.x': 0.22; '64-bit': 0.22; 'arrays': 0.22; 'see:': 0.22; 'skip:= 20': 0.22; "user's": 0.22; '(like': 0.23; 'performing': 0.23; 'tables': 0.23; 'unix': 0.24; 'feature': 0.24; 'install': 0.25; 'appear': 0.26; 'addition,': 0.27; 'earlier': 0.27; '----': 0.27; 'least': 0.27; 'message-id:@mail.gmail.com': 0.27; '(although': 0.29; '32-bit': 0.29; 'declared': 0.29; 'fast.': 0.29; 'for,': 0.29; 'i/o': 0.29; 'cc:no real name:2**1': 0.29; 'objects': 0.29; 'typically': 0.29; 'allows': 0.30; 'minimal': 0.30; 'query': 0.30; 'operations': 0.31; 'probably': 0.31; 'at:': 0.31; 'platforms.': 0.33; 'received:google.com': 0.35; 'so,': 0.35; 'ones': 0.35; 'next': 0.35; 'installing': 0.35; 'list:': 0.35; 'stable': 0.35; 'expected': 0.35; 'supports': 0.35; 'but': 0.36; 'should': 0.36; 'project': 0.36; 'url:org': 0.36; 'received:209.85': 0.36; 'possible': 0.36; '(and': 0.36; 'data.': 0.36; 'faster': 0.36; 'framework': 0.36; 'notes': 0.36; 'operating': 0.37; 'release': 0.37; 'doing': 0.38; 'exciting': 0.38; 'received:209': 0.38; 'several': 0.38; 'files': 0.38; 'means': 0.39; 'subject:[': 0.39; 'test': 0.39; 'data': 0.39; 'format': 0.39; 'url:en': 0.39; 'resources': 0.39; 'some': 0.40; 'future': 0.60; 'your': 0.60; 'url:group': 0.61; 'default': 0.61; 'real': 0.62; 'here.': 0.62; 'more': 0.63; 'visit': 0.64; 'license': 0.65; 'series': 0.65; 'promise': 0.66; 'below.': 0.66; 'decided': 0.66; 'header:Reply- To:1': 0.67; 'home': 0.67; 'finally': 0.70; 'reply-to:no real name:2**0': 0.71; 'capabilities': 0.72; 'reply-to:addr:gmail.com': 0.76; 'carried': 0.76; 'removal': 0.79; 'column.': 0.84; 'lastly,': 0.84; 'pip': 0.84; 'received:209.85.160.178': 0.84; 'together,': 0.84; 'url:master': 0.84; 'url:readthedocs': 0.84; 'announcing': 0.91; 'beneficial': 0.91; 'generation.': 0.91; 'url:latest': 0.91; 'reducing': 0.93; 'url:url': 0.93; 'assured': 0.95; 'candidate,': 0.95; 'email addr:googlegroups.com': 0.95 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:date:message-id:subject:from:to:cc; bh=j+Q1oL7TfGszWYRwh3DqCayX1GxyTxj8eGSMadD9o8Q=; b=HBi6QnufO4D9p6NLcc2ZJWHd43RLA3AxrxjmppxfS0vBZmrYz08FgsK6Z90QHc2ob3 plc6PI4R0WV6iGC4OEreJ4OTHaNxsOFzXuyvsjUzW0t6RRrRhOEqK53wnHOdzNcsorCs ciUfVAJxyXMmcEvxi3WFm/mhvJDJXF/VU+nD+CyfhF8VxeJfFBor5DsUYKEAdOFxBG5L 1tz57QacLCU1PNFiJATn691hPFNUNYAJPI2Wx90/RicQFYUNZSqY5uKd/s9b2V0Gnsv5 X5c2T8h7jzn1tefBnbYPojE+wGS5y7Nph1ov7I5eHfeGoCbujrXhyTu7PMpkLlutNcw1 6cLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:reply-to:date:message-id:subject :from:to:cc; bh=j+Q1oL7TfGszWYRwh3DqCayX1GxyTxj8eGSMadD9o8Q=; b=Hctjd0/mUEg8PCtUb53flyT1fGshI9ypYJaVl1HlOid46GvmoisJEQ1Ei75Qg3Myl0 wbHANtj7ugieCvTAmAgt5YaRizT5SjBkds5xXWsY/Md2cHqOC1sAr0H+sG9l1oTh4NAS L0Yb0CDIwG7YrWFcToTGldRb8OixUJ+UFVXpASzB5YNjlM/+Ur5vS2Er+3IrcRIbhcsn kPydTUUzAohaBBUsMJslgtBFJ0OH8Dege1vXG+U9PYlyyQDr/WwNx7my3hI5Y6Y0VD9d cbx70IaM+GNENasYS4yn+Kmx3g5m+jAnnWtiMFZ4G3BLr6PF2dD8sS+eD8PjAKpQX6qI e5uQ== X-Gm-Message-State: AD7BkJIV9KR1JO+CDDdomvqXh17Mn6HUL/UMhTWwqjI3so9LR9/+uWOl1VRlAqZPPbXhJaEFPIHN+OGsZ7pWDg== X-Received: by 10.37.231.151 with SMTP id e145mr5332721ybh.148.1457443665024; Tue, 08 Mar 2016 05:27:45 -0800 (PST) X-Mailman-Approved-At: Tue, 08 Mar 2016 08:28:09 -0500 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: python-announce-list@python.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Announcement-only list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Xref: csiph.com comp.lang.python.announce:2060 ========================== Announcing bcolz 1.0.0 RC1 ========================== What's new ========== Yeah, 1.0.0 is finally here. We are not introducing any exciting new feature (just some optimizations and bug fixes), but bcolz is already 6 years old and it implements most of the capabilities that it was designed for, so I decided to release a 1.0.0 meaning that the format is declared stable and that people can be assured that future bcolz releases will be able to read bcolz 1.0 data files (and probably much earlier ones too) for a long while. Such a format is fully described at: https://github.com/Blosc/bcolz/blob/master/DISK_FORMAT_v1.rst Also, a 1.0.0 release means that bcolz 1.x series will be based on C-Blosc 1.x series (https://github.com/Blosc/c-blosc). After C-Blosc 2.x (https://github.com/Blosc/c-blosc2) would be out, a new bcolz 2.x is expected taking advantage of shiny new features of C-Blosc2 (more compressors, more filters, native variable length support and the concept of super-chunks), which should be very beneficial for next bcolz generation. Important: this is a Release Candidate, so please test it as much as you can. If no issues would appear in a week or so, I will proceed to tag and release 1.0.0 final. Enjoy! For a more detailed change log, see: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/querying-ep14.ipynb#Plots Other users of bcolz are Visualfabriq (http://www.visualfabriq.com/) the Blaze project (http://blaze.pydata.org/), Quantopian (https://www.quantopian.com/) and Scikit-Allel (https://github.com/cggh/scikit-allel) which you can read more about by pointing your browser at the links below. * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Blaze: * Notebooks showing Blaze + Pandas + BColz interaction: * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-csv.ipynb * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-bcolz.ipynb * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html * Scikit-Allel * Provides an alternative backend to work with compressed arrays * https://scikit-allel.readthedocs.org/en/latest/model/bcolz.html Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz@googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!** -- Francesc Alted